Skip to content

Release v1.63.0 - #7011

Merged
neubig merged 2 commits into
mainfrom
rel-1.63.0
Nov 19, 2026
Merged

neubig merged 2 commits into
mainfrom
rel-1.63.0

Conversation

@openhands-release-bot

@openhands-release-bot openhands-release-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Release v1.63.0

This PR prepares the release for version 1.63.0.

Started by: @neubig

Release Checklist

  • Version set to 1.63.0
  • Temporary .pr/ artifacts removed
  • Fix any deprecation deadlines if they exist
  • Integration tests pass (tagged with integration-test)
  • Behavior tests pass (tagged with behavior-test)
  • Example tests pass (tagged with test-examples)
  • Security scan passes (tagged with security-scan)
  • Confirm any release-note-required PRs are accurately called out in the final release notes

Updating this branch

If main advances, refresh this branch as described in .github/workflows/README-RELEASE.md instead of using Update branch, so new .pr/ artifacts are removed in the same update.

What happens on merge

When this PR is merged, the create-release.yml workflow will automatically:

  1. Create a GitHub release with tag v1.63.0 and auto-generated notes, plus an explicit preamble for merged release-note-required PRs
  2. Explicitly dispatch PyPI, npm, GitHub Packages, agent-server image, and binary publishers against tag v1.63.0
  3. Trigger version-bump-prs.yml after successful PyPI publication

🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:010400e-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-010400e-python \
  ghcr.io/openhands/agent-server:010400e-python

All tags pushed for this build

ghcr.io/openhands/agent-server:010400e-golang-amd64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-golang-amd64
ghcr.io/openhands/agent-server:rel-1.63.0-golang-amd64
ghcr.io/openhands/agent-server:010400e-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:010400e-golang-arm64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-golang-arm64
ghcr.io/openhands/agent-server:rel-1.63.0-golang-arm64
ghcr.io/openhands/agent-server:010400e-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:010400e-java-amd64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-java-amd64
ghcr.io/openhands/agent-server:rel-1.63.0-java-amd64
ghcr.io/openhands/agent-server:010400e-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:010400e-java-arm64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-java-arm64
ghcr.io/openhands/agent-server:rel-1.63.0-java-arm64
ghcr.io/openhands/agent-server:010400e-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:010400e-python-amd64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python-amd64
ghcr.io/openhands/agent-server:rel-1.63.0-python-amd64
ghcr.io/openhands/agent-server:010400e-python-node-runtime-amd64
ghcr.io/openhands/agent-server:010400e-python-arm64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python-arm64
ghcr.io/openhands/agent-server:rel-1.63.0-python-arm64
ghcr.io/openhands/agent-server:010400e-python-node-runtime-arm64
ghcr.io/openhands/agent-server:010400e-python-minimal-amd64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python-minimal-amd64
ghcr.io/openhands/agent-server:rel-1.63.0-python-minimal-amd64
ghcr.io/openhands/agent-server:010400e-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:010400e-python-minimal-arm64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python-minimal-arm64
ghcr.io/openhands/agent-server:rel-1.63.0-python-minimal-arm64
ghcr.io/openhands/agent-server:010400e-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:010400e-python-slim-amd64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python-slim-amd64
ghcr.io/openhands/agent-server:rel-1.63.0-python-slim-amd64
ghcr.io/openhands/agent-server:010400e-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:010400e-python-slim-arm64
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python-slim-arm64
ghcr.io/openhands/agent-server:rel-1.63.0-python-slim-arm64
ghcr.io/openhands/agent-server:010400e-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:010400e-golang
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-golang
ghcr.io/openhands/agent-server:rel-1.63.0-golang
ghcr.io/openhands/agent-server:010400e-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:010400e-java
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-java
ghcr.io/openhands/agent-server:rel-1.63.0-java
ghcr.io/openhands/agent-server:010400e-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:010400e-python-minimal
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python-minimal
ghcr.io/openhands/agent-server:rel-1.63.0-python-minimal
ghcr.io/openhands/agent-server:010400e-python-node-runtime-minimal
ghcr.io/openhands/agent-server:010400e-python-slim
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python-slim
ghcr.io/openhands/agent-server:rel-1.63.0-python-slim
ghcr.io/openhands/agent-server:010400e-python-node-runtime-slim
ghcr.io/openhands/agent-server:010400e-python
ghcr.io/openhands/agent-server:010400ee07a800ea3c7b27548d1a8ef62238f4ee-python
ghcr.io/openhands/agent-server:rel-1.63.0-python
ghcr.io/openhands/agent-server:010400e-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 010400e-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 010400e-python-amd64) are also available if needed

Co-authored-by: openhands <openhands@all-hands.dev>
@openhands-release-bot openhands-release-bot Bot added integration-test Runs the integration tests and comments the results test-examples Run all applicable "examples/" files. Expensive operation. behavior-test security-scan Run the release security scan (approval-drift + supply-chain) on this PR labels Nov 19, 2026
@neubig

neubig commented Nov 19, 2026

Copy link
Copy Markdown
Member

AI agent (OpenHands) acting on behalf of Graham Neubig: I triggered the official Prepare Release workflow to propagate the merged filelock remediation (#6811) into a released SDK and downstream automation image. I am reviewing this generated release PR and its integration, behavior, example, security, and compatibility checks before approval or merge. No checks will be bypassed.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown
Contributor

Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly.

@github-actions

Copy link
Copy Markdown
Contributor

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

@github-actions

Copy link
Copy Markdown
Contributor

🔒 Release Security Scan

🔒 Approval drift (time-of-check vs time-of-use)

❌ 29 finding(s)

Baseline: v1.62.0 — 127 merged PR(s) in range.

⚠️ 1 first-parent commit(s) in range did not match the (#N) squash convention and were not audited (a merge/rebase merge, or a lost (#N) suffix). These are a blind spot — inspect them:

  • Release v1.63.0
PR finding detail
#6960 changed-after-approval last approval by @all-hands-bot was on f963a03, but merged head was 4a8c78dcf — commits landed after review
#6547 changed-after-approval last approval by @enyst was on e066e4d, but merged head was 9d63b9d88 — commits landed after review
#6901 changed-after-approval last approval by @all-hands-bot was on fa3c56d, but merged head was 72ff2894a — commits landed after review
#6876 changed-after-approval last approval by @all-hands-bot was on 4cc3c3d, but merged head was ab1ad1035 — commits landed after review
#6791 changed-after-approval last approval by @all-hands-bot was on 33a5b06, but merged head was 02d02fe73 — commits landed after review
#6851 changed-after-approval last approval by @all-hands-bot was on 598f47b, but merged head was c88779bf4 — commits landed after review
#6892 changed-after-approval last approval by @all-hands-bot was on e67fdff, but merged head was eecd315d0 — commits landed after review
#6881 changed-after-approval last approval by @all-hands-bot was on 62fdf22, but merged head was c0f5c5d14 — commits landed after review
#6852 changed-after-approval last approval by @all-hands-bot was on c5239d7, but merged head was e2da143e7 — commits landed after review
#6880 changed-after-approval last approval by @all-hands-bot was on 451fd59, but merged head was 1f260adee — commits landed after review
#6879 changed-after-approval last approval by @all-hands-bot was on 11da64c, but merged head was 1f4ca7086 — commits landed after review
#6877 changed-after-approval last approval by @all-hands-bot was on b188f2e, but merged head was daa0c3d9a — commits landed after review
#6874 changed-after-approval last approval by @all-hands-bot was on 216e353, but merged head was 5b310bfce — commits landed after review
#6872 changed-after-approval last approval by @all-hands-bot was on 35bfa30, but merged head was b05a590b8 — commits landed after review
#6871 changed-after-approval last approval by @all-hands-bot was on 52b8bce, but merged head was 427c57d48 — commits landed after review
#6870 changed-after-approval last approval by @all-hands-bot was on d6c54f1, but merged head was 4287f657e — commits landed after review
#6781 changed-after-approval last approval by @all-hands-bot was on 8f9eab8, but merged head was 0332a541f — commits landed after review
#6779 changed-after-approval last approval by @enyst was on 3001466, but merged head was 6b5972678 — commits landed after review
#6822 changed-after-approval last approval by @all-hands-bot was on 5182777, but merged head was bd01ae3ae — commits landed after review
#6823 changed-after-approval last approval by @all-hands-bot was on 6e38e2c, but merged head was 17da74b0c — commits landed after review
#6753 changed-after-approval last approval by @all-hands-bot was on 1315463, but merged head was e6510630d — commits landed after review
#5776 changed-after-approval last approval by @VascoSch92 was on 5b985cf, but merged head was 7a46b3c90 — commits landed after review
#5815 changed-after-approval last approval by @all-hands-bot was on ad4cb63, but merged head was 33b66918b — commits landed after review
#6040 changed-after-approval last approval by @all-hands-bot was on edb841b, but merged head was 145fee55f — commits landed after review
#6668 changed-after-approval last approval by @VascoSch92 was on 9302afd, but merged head was 729fa8f37 — commits landed after review
#5757 changed-after-approval last approval by @all-hands-bot was on fa2680e, but merged head was ad847c2a7 — commits landed after review
#5756 changed-after-approval last approval by @VascoSch92 was on 1df2cf0, but merged head was 270e12c21 — commits landed after review
#5524 changed-after-approval last approval by @all-hands-bot was on efca744, but merged head was 50f00032c — commits landed after review
#6609 changed-after-approval last approval by @all-hands-bot was on d5bf5ef, but merged head was 82d61a25b — commits landed after review

Audited 127 PR(s): 98 clean, 29 flagged, 0 un-auditable.

📦 Supply-chain dependency diff

✅ no findings

Baseline: v1.62.0
Added: 1, bumped: 30, removed: 0.
(plus 4 internal openhands-* bump(s))

OSV: no known vulns across 35 new/bumped dep(s).

Added dependencies
  • python-discovery==1.9.5
Bumped dependencies
  • agent-client-protocol: 0.12.1 → 1.0.0
  • aiofile: 3.9.0 → 3.12.3
  • authlib: 1.6.12 → 1.9.5
  • boto3: 1.43.123 → 1.44.1
  • botocore: 1.43.123 → 1.44.1
  • cachetools: 7.2.8 → 8.0.0
  • caio: 0.9.25 → 0.12.10
  • distlib: 0.4.0 → 0.4.4
  • fastmcp: 4.1.4 → 4.1.8
  • fastmcp-slim: 4.1.4 → 4.1.8
  • filelock: 3.20.3 → 4.2.9
  • google-cloud-aiplatform: 2.7.0 → 2.8.0
  • griffe: 2.3.0 → 2.4.1
  • griffecli: 2.3.0 → 2.4.1
  • griffelib: 2.3.0 → 2.4.1
  • joserfc: 1.7.7 → 1.8.0
  • litellm: 1.105.1 → 1.108.3
  • posthog: 7.82.2 → 7.86.4
  • pre-commit: 4.3.0 → 4.7.0
  • py-key-value-aio: 0.4.4 → 0.4.6
  • pydantic: 2.13.5 → 2.14.3
  • pydantic-core: 2.46.5 → 2.50.4
  • pytest-asyncio: 1.3.0 → 1.4.1.post1
  • ruff: 0.17.3 → 0.17.4
  • sqlalchemy: 2.1.6 → 2.1.8
  • streamlit: 1.66.1 → 1.67.0
  • tree-sitter: 0.25.2 → 0.26.1
  • typing-extensions: 4.15.0 → 4.16.0
  • typing-inspection: 0.4.2 → 0.4.5
  • virtualenv: 20.36.1 → 21.15.6
Internal `openhands-*` bumps
  • openhands-agent-server: 1.62.0 → 1.63.0
  • openhands-sdk: 1.62.0 → 1.63.0
  • openhands-tools: 1.62.0 → 1.63.0
  • openhands-workspace: 1.62.0 → 1.63.0

Deterministic scanners: approval-drift + supply-chain dependency diff. Read-only; no PR code executed.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base main@bc66f71b39d76876e48a50c5d0884590aed8249b vs this PR merge 418a01aa15, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.188 s [0.174, 0.195] 0.186 s [0.158, 0.201] -1% ≈ same
↳ first agent event 0.255 s [0.227, 0.261] 0.246 s [0.216, 0.278] -3% ≈ same
First create right after server start 0.403 s [0.311, 0.427] 0.394 s [0.261, 0.487] -2% ≈ same
↳ first agent event 0.750 s [0.546, 0.867] 0.780 s [0.458, 0.841] +4% ≈ same
First create 3 s after server start 0.174 s [0.147, 0.210] 0.164 s [0.138, 0.228] -6% ≈ same
↳ first agent event 0.510 s [0.367, 0.639] 0.490 s [0.334, 0.585] -4% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.698 s [0.664, 0.710] 0.708 s [0.697, 0.725] +1% ≈ same
↳ first agent event 0.749 s [0.709, 0.768] 0.767 s [0.745, 0.787] +2% ≈ same
Worktree creates that stalled the server > 20 ms 1/20 (5%) 1/20 (5%) +0 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.155 s [1.135, 1.180] 1.156 s [1.145, 1.178] +0% ≈ same
↳ first agent event 3.921 s [3.838, 4.079] 3.909 s [3.845, 4.057] -0% ≈ same
Creates that stalled the server > 20 ms 2/35 (6%) 2/35 (6%) +0 pts ≈ same
Server startup (/alive) 5.143 s [4.451, 5.476] 5.059 s [4.751, 5.772] -2% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-7gjsq5q4noihfj7y, 176 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

@neubig

neubig commented Nov 19, 2026

Copy link
Copy Markdown
Member

AI agent (OpenHands) acting on behalf of Graham Neubig: release is on hold. The current-head Release Security Scan failed its approval-drift gate: 29 of 127 merged PRs differ from their approved head; the dependency-diff check passed. Independent review confirms the generated version/schema/lock updates and inclusion of #6811 (filelock 4.2.9) and #6925, but recommends HOLD pending gate reconciliation and completion of integration/behavior/example checks. I have not approved or merged this release and will not waive the gate. I am classifying post-approval deltas versus main-merge/rebase history to provide actionable evidence rather than assuming all findings represent unsafe code.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
TOTAL57741930584% 
report-only-changed-files is enabled. No changed files were found in the coverage report :)

@neubig

neubig commented Nov 19, 2026

Copy link
Copy Markdown
Member

AI agent (OpenHands) acting on behalf of Graham Neubig: blocker diagnosis is now narrower. Both integration and behavior matrices fail before model calls with ModuleNotFoundError: No module named tests.integration; the workflow invokes the package-importing runner by file path. A local module-invocation fix with CLI smoke regressions is in preparation. Approval-drift triage classifies the 29 mismatches as 12 direct-edit, 11 conflict-resolution, 4 inherited/stack-only, and 2 patch-equivalent-rebase cases—not 29 independent new feature edits. There are nevertheless substantive post-approval integrations, and the current gate has no supported attestation/equivalence reconciliation. Release remains blocked; no approvals, policy exceptions, or release merges performed. Maintainer/security-owner reconciliation is needed before rerunning that gate.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

🔄 Running Examples with openhands/claude-haiku-4-5-20251001

Generated: 2026-11-19 18:16:17 UTC

Example Status Duration Cost
01_standalone_sdk/02_custom_tools.py ✅ PASS 32.8s $0.03
01_standalone_sdk/03_activate_skill.py ✅ PASS 21.2s $0.03
01_standalone_sdk/05_use_llm_registry.py ✅ PASS 8.2s $0.00
01_standalone_sdk/07_mcp_integration.py ✅ PASS 30.4s $0.02
01_standalone_sdk/09_pause_example.py ✅ PASS 16.4s $0.01
01_standalone_sdk/10_persistence.py ✅ PASS 26.0s $0.04
01_standalone_sdk/11_async.py ✅ PASS 25.9s $0.03
01_standalone_sdk/12_custom_secrets.py ✅ PASS 14.0s $0.01
01_standalone_sdk/13_get_llm_metrics.py ✅ PASS 24.6s $0.03
01_standalone_sdk/14_context_condenser.py ✅ PASS 4m 32s $0.00
01_standalone_sdk/17_image_input.py ✅ PASS 22.5s $0.02
01_standalone_sdk/18_send_message_while_processing.py ✅ PASS 17.4s $0.01
01_standalone_sdk/19_llm_routing.py ✅ PASS 18.1s $0.02
01_standalone_sdk/20_stuck_detector.py ✅ PASS 14.8s $0.02
01_standalone_sdk/21_generate_extraneous_conversation_costs.py ✅ PASS 10.9s $0.01
01_standalone_sdk/22_anthropic_thinking.py ✅ PASS 16.7s $0.01
01_standalone_sdk/23_responses_reasoning.py ✅ PASS 43.9s $0.01
01_standalone_sdk/24_planning_agent_workflow.py ✅ PASS 4m 34s $0.26
01_standalone_sdk/25_agent_delegation.py ✅ PASS 1m 10s $0.08
01_standalone_sdk/26_custom_visualizer.py ✅ PASS 20.0s $0.02
01_standalone_sdk/28_ask_agent_example.py ✅ PASS 33.1s $0.04
01_standalone_sdk/29_llm_streaming.py ✅ PASS 29.0s $0.02
01_standalone_sdk/30_tom_agent.py ✅ PASS 34.7s $0.01
01_standalone_sdk/31_iterative_refinement.py ✅ PASS 1m 34s $0.07
01_standalone_sdk/32_configurable_security_policy.py ✅ PASS 18.0s $0.02
01_standalone_sdk/33_hooks/main.py ✅ PASS 31.4s $0.04
01_standalone_sdk/34_critic_example.py ✅ PASS 25.8s $0.02
01_standalone_sdk/36_event_json_to_openai_messages.py ✅ PASS 16.4s $0.00
01_standalone_sdk/37_llm_profile_store/main.py ✅ PASS 6.8s $0.00
01_standalone_sdk/38_browser_session_recording.py ✅ PASS 50.1s $0.03
01_standalone_sdk/39_llm_fallback.py ✅ PASS 21.2s $0.01
01_standalone_sdk/40_acp_agent_example.py ✅ PASS 41.6s $0.20
01_standalone_sdk/41_task_tool_set.py ✅ PASS 36.1s $0.03
01_standalone_sdk/42_file_based_subagents.py ✅ PASS 30.4s $0.03
01_standalone_sdk/44_model_switching_in_convo.py ✅ PASS 11.2s $0.01
01_standalone_sdk/45_parallel_tool_execution.py ✅ PASS 3m 17s $0.40
01_standalone_sdk/46_agent_settings.py ✅ PASS 12.0s $0.01
01_standalone_sdk/47_defense_in_depth_security.py ✅ PASS 3.0s $0.00
01_standalone_sdk/48_conversation_fork.py ✅ PASS 18.7s $0.01
01_standalone_sdk/49_switch_llm_tool.py ✅ PASS 9.9s $0.04
01_standalone_sdk/50_async_cancellation.py ✅ PASS 12.8s $0.00
01_standalone_sdk/51_agent_hooks/main.py ✅ PASS 38.8s $0.04
01_standalone_sdk/52_dynamic_workflow.py ✅ PASS 2m 29s $0.12
01_standalone_sdk/53_client_defined_tools.py ✅ PASS 14.3s $0.01
01_standalone_sdk/54_goal_completion_loop.py ✅ PASS 29.4s $0.03
01_standalone_sdk/55_persistent_memory.py ✅ PASS 17.1s $0.02
01_standalone_sdk/56_structured_output.py ✅ PASS 51.2s $0.07
01_standalone_sdk/57_prompt_hooks/main.py ✅ PASS 14.8s $0.00
01_standalone_sdk/58_ask_oracle_tool/main.py ✅ PASS 13.6s $0.01
01_standalone_sdk/59_route_task_to_model.py ✅ PASS 14.3s $0.04
02_remote_agent_server/01_convo_with_local_agent_server.py ✅ PASS 43.1s $0.02
02_remote_agent_server/02_convo_with_docker_sandboxed_server.py ✅ PASS 1m 15s $0.04
02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py ✅ PASS 2m 49s $0.11
02_remote_agent_server/04_convo_with_api_sandboxed_server.py ❌ FAIL
Exit code 1
8.5s --
02_remote_agent_server/06_custom_tool/main.py ✅ PASS 5m 14s $0.06
02_remote_agent_server/07_convo_with_cloud_workspace.py ✅ PASS 40.9s $0.02
02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py ✅ PASS 4m 26s $0.03
02_remote_agent_server/09_acp_agent_with_remote_runtime.py ❌ FAIL
Exit code 1
6.0s --
02_remote_agent_server/10_cloud_workspace_share_credentials.py ✅ PASS 38.8s $0.00
02_remote_agent_server/11_conversation_fork.py ✅ PASS 41.5s $0.00
02_remote_agent_server/12_settings_and_secrets_api.py ✅ PASS 2m 32s $0.02
02_remote_agent_server/13_workspace_get_llm.py ✅ PASS 44.6s $0.01
02_remote_agent_server/14_client_defined_tools.py ✅ PASS 36.9s $0.02
02_remote_agent_server/15_openai_compatible_gateway.py ✅ PASS 21.1s $0.04
02_remote_agent_server/16_deferred_init.py ✅ PASS 20.6s $0.01
04_llm_specific_tools/01_gpt5_apply_patch_preset.py ✅ PASS 19.9s $0.02
04_llm_specific_tools/02_gemini_file_tools.py ✅ PASS 36.4s $0.11
05_skills_and_plugins/01_loading_agentskills/main.py ✅ PASS 13.9s $0.02
05_skills_and_plugins/02_loading_plugins/main.py ✅ PASS 19.4s $0.03
05_skills_and_plugins/04_mixed_marketplace_skills/main.py ✅ PASS 4.6s $0.00

❌ Some tests failed

Total: 70 | Passed: 68 | Failed: 2 | Total Cost: $2.55

Failed examples:

  • examples/02_remote_agent_server/04_convo_with_api_sandboxed_server.py: Exit code 1
  • examples/02_remote_agent_server/09_acp_agent_with_remote_runtime.py: Exit code 1

View full workflow run

@neubig

neubig commented Nov 19, 2026

Copy link
Copy Markdown
Member

AI agent (OpenHands) acting on behalf of Graham Neubig: pushed the integration-runner fix in 010400e. Workflow entrypoints now use python -m tests.integration...; usage examples are synchronized and a regression exercises all four actual workflow commands with --help and no PYTHONPATH. Verification: 4 regression tests passed; Ruff lint/format and git diff --check passed. No LLM calls made by these regressions. Investigation artifacts were not committed. This fixes runner startup, not the separate approval-drift gate. Label-triggered pull_request_target runs use the base workflow, so rerunning the old runs will not exercise this workflow change; validation of the fixed workflow must select the release branch explicitly.

@neubig

neubig commented Nov 19, 2026

Copy link
Copy Markdown
Member

AI agent (OpenHands) acting on behalf of Graham Neubig.

Confirmed cause of integration/behavior runner startup failure

The LiteLLM upgrade in #6811 (1.105.1 → 1.108.3, needed for the filelock security fix) exposed an existing import-path dependency in our test invocation.

In LiteLLM 1.105.1, litellm/__init__.py imports proxy.proxy_cli, whose module-level code executes sys.path.append(os.getcwd()). That accidentally makes the repository root importable when we run python tests/integration/run_infer.py. LiteLLM 1.108.3 removes that automatic proxy CLI import, so the runner can no longer resolve tests.integration.

I reproduced this using official PyPI wheels, holding the SDK code, interpreter, and other installed dependencies constant and selecting only the LiteLLM version:

Invocation LiteLLM Result
python tests/integration/run_infer.py --help 1.105.1 Pass
Same file-path invocation 1.108.3 ModuleNotFoundError: No module named tests.integration
python -m tests.integration.run_infer --help 1.108.3 Pass

A traced import also confirmed the old proxy CLI appends the working directory; the new version does not. This isolates the trigger rather than attributing it to model behavior or a transient runner failure.

The fix already pushed in 010400e uses module invocation, eliminating reliance on this third-party side effect while retaining the security upgrade. All four workflow-entrypoint regression tests pass locally. These are startup-only tests with no LLM calls; full integration/behavior validation on the fixed head is still needed. This diagnosis does not resolve the separate approval-drift security gate.

@neubig
neubig merged commit 98f0fe1 into main Nov 19, 2026
55 checks passed
@neubig
neubig deleted the rel-1.63.0 branch November 19, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

behavior-test integration-test Runs the integration tests and comments the results security-scan Run the release security scan (approval-drift + supply-chain) on this PR test-examples Run all applicable "examples/" files. Expensive operation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants