Skip to content

feat(evaluator): run sequential judges in the shared case runtime - #315

Open
zpzjzj wants to merge 4 commits into
mainfrom
feat/docker-judge-snapshots
Open

zpzjzj wants to merge 4 commits into
mainfrom
feat/docker-judge-snapshots

Conversation

@zpzjzj

@zpzjzj zpzjzj commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Multi-judge evaluations previously required filesystem snapshot/fork support. Run the evaluated Agent once, then execute named judges sequentially in its existing runtime, following shared-verifier semantics. Each Agent judge starts a fresh session; verdicts, timeouts and archived materials remain separate by judge ID.

Filesystem changes, installed tools, services, volumes, environment and network policy remain shared. A completed judge FAIL or ERROR does not prevent later judges from running; case cancellation skips pending members. Remove the obsolete judge snapshot capability and workspace-copy implementation, retaining the normal case cleanup policy.

Stacked on #314 to preserve independent judge engine validation and provisioning. Part of #308; filesystem-isolation-specific acceptance remains outside this shared-runtime change, so #308 stays open.

Validation:

  • make fmt, make verify, make test with race detection, make build, and make e2e.
  • Regression coverage for one Agent run, fresh sessions without cross-member resume, shared mutations, independent engines, gate/timeout/cancellation outcomes and namespaced artifact readback.
  • Real Docker shared-runtime tests: all members PASS, and script FAIL / both Agent judges PASS / aggregate FAIL.
  • Live OpenSandbox shared-runtime tests in stacked test(runtime): verify sequential judges in a shared OpenSandbox #316 exercise the same runtime path. The shared-environment fixture is deterministic; an additional test(runtime): verify sequential judges in a shared OpenSandbox #316 test runs the real OpenCode CLI and makes live model-provider requests.

The stack includes #314's independent OpenCode provider initialization fix and its endpoint/API-key/MCP preservation regressions.

Default-endpoint follow-up:

  • Independent OpenCode judges reset only their selected provider's inherited inline configuration, including endpoint and API-key/SDK/model settings; case MCP, unrelated providers, plugins and permissions remain intact. Normal runner and legacy judge initialization retain their existing behavior.
  • Regression uses the real resolver and adapter factory. Both OpenAI and Anthropic omitted-base_url cases reproduce on the previous evaluator and pass after the fix; explicit-endpoint controls pass as well.
  • Real OpenSandbox/OpenCode 1.14.24 default-configuration installation, preflight and readback passed. The real qwen3.8-max explicit-endpoint control passed with two distinct judge sessions.
  • Native OpenAI model calls were not completed: direct api.openai.com connectivity timed out from the test sandbox, and both model invocations hit their deadlines. No native OpenAI model success is claimed.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T07:44:55.544167Z 3796e6e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@zpzjzj
zpzjzj force-pushed the feat/docker-judge-snapshots branch from 3796e6e to 7bed79b Compare October 13, 2026 09:50
@zpzjzj zpzjzj changed the title feat(runtime): isolate Docker judges with filesystem snapshots feat(evaluator): run sequential judges in the shared case runtime Oct 13, 2026
@zpzjzj
zpzjzj force-pushed the feat/docker-judge-snapshots branch from f3aa2b8 to 6ca4777 Compare October 14, 2026 08:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant