Skip to content

feat(task-steps): native OSWorld initialize and evaluate on computer envs (FD-5492) - #606

Open
williamloo-scale wants to merge 9 commits into
mainfrom
wloo/osworld-native-computer-steps
Open

williamloo-scale wants to merge 9 commits into
mainfrom
wloo/osworld-native-computer-steps

Conversation

@williamloo-scale

@williamloo-scale williamloo-scale commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Please read this Claude Artifact https://claude.ai/artifact/JCkogrfdgTxvBhXLM2PoZS?sk=kv0m_DF3_tK47coA7YHyfQ

Note

Status after review: the controller these steps deploy is built from private scale-cua, so this PR doesn't merge to core until the controller and guest server are public (see Edgar's first comment). Meanwhile the same steps ship as an agentenvhub-sdk plugin for Scale, in scaleapi/scaleapi#175341, which unblocks the hub (scaleapi/scaleapi#174344). Both copies have the review fixes (20fdec6, ab7db64).

Summary

This PR adds three task steps to core: osworld_deploy_controller, osworld_initialize and osworld_evaluate. It also adds the osworld-server computer app (all in src/agent_env/task_step/task_steps/computer/osworld/, beside the other computer steps). Together they let an OSWorld task run on a framework computer env. Both task kinds are supported:

  • Classic: init_config plus evaluator.
  • OSWorld-V2: a task_NNN.py that defines TASK_CLASS.

No plugin is needed.

Important

Merge order: scaleapi/scale-cua#360 must merge, and a controller image built from it must be published, before this PR. This PR refuses any controller that doesn't enforce CONTROLLER_AUTH_TOKEN, and today's image doesn't.

Design

The controller runs as a separate sidecar.

  • The OSWorld controller holds the grader, the task's checks and the judge's model key.
  • A computer's apps run as its session user, and the agent runs as that user too.
  • So the controller runs in a container sandbox of its own, beside the computer, in remote mode. The agent can't read it.

Only the framework may call the controller.

  • The controller's port can be reachable from the computer, so without a check the agent could call /setup or /evaluate itself.
  • The step gives the controller CONTROLLER_AUTH_TOKEN, and every call to the controller carries it as a bearer token. The controller answers 401 to any call without it (scale-cua#360).
  • The token is HMAC-SHA256 of the env's control token over osworld-controller:<env_id>. It's derived again whenever it's needed and never stored, because context redaction strips stored secrets.
  • Before recording the controller, the step sends a call without the token and refuses the controller unless it gets 401.

The computer runs only the OSWorld guest server, as the osworld-server app.

  • The app's own server forwards /osworld/<route> to 127.0.0.1:5000.
  • Calls need the env's control token, which the app removes before forwarding.

The guest server is found through the computer's card.

  • The app's card advertises the extension urn:agentenv:osworld-guest/v1 at its forward prefix. A GET on that endpoint reports whether the guest server is up.
  • The computer roots the endpoint wherever it serves the app. So osworld_deploy_controller finds the guest server by the extension's URI in the computer's card, and no /svc/<name> route is built anywhere.
  • It reads the stored card first. If the app's server wasn't answering yet when that card was read, it reads the live card again for up to 2 minutes.
  • A computer without the app fails before a sandbox is created.

Step reference

osworld_deploy_controller

  • Fields: env_id, controller_image_artifact_id (a docker_image artifact; default cua-controller), ttl_seconds, cpu, memory_mb, env_vars.
  • Controller env vars:
    • PROVIDER=remote, OS_TYPE=Ubuntu, REMOTE_VM_IP=<guest URL from the card>, and REMOTE_SERVER_HEADERS=<JSON of DeployedEnv.control_headers>;
    • CONTROLLER_AUTH_TOKEN=<the derived token>;
    • LITELLM_API_KEY and LITELLM_BASE_URL from [model], when they're configured.
    • env_vars go first, so the step's own values win.
    • Only the variable names are logged.
  • Sandbox: 10 GB of disk.
    • With a chain provider, each link gets its own guest and controller addresses (url_from_sandbox per link), and prepare_image runs on that link before create_container.
    • If a link fails, or its sandbox has a problem, the sandbox is terminated and the next link is tried. If every link fails, the step fails and lists why.
  • Waits until GET /configure answers (the controller has no health route), then checks that the controller requires the token.
  • Records on the env (DeployedEnv.metadata): osworld_controller_url, osworld_controller_sandbox_id and osworld_guest_url.
  • Teardown: the sandbox goes into context.deployed_sandboxes as osworld-controller-<env_id>.

osworld_initialize

  • Fields: env_id, init_config, osworld_v2_task_path, and the legacy osworld_v2_task_url (the path wins if both are set).
  • OSWorld-V2: POST /reset, then POST /initialize_task {id, osworld_v2_task_url}. It fails on success: false or a truthy error.
  • Classic: sends the whole init_config to POST /setup, which serves every setup type. It fails on success: false or on any seeded site with ok: false. Seeded sites are recorded in context.metadata["osworld_seeded_sites"].
  • The controller's error text in a failure is cut to 300 characters.

osworld_evaluate

  • Fields: env_id, evaluator, verifier_id (default: a uuid hex), osworld_v2_task_path / osworld_v2_task_url, and judge_model.
  • OSWorld-V2: /evaluate {osworld_v2_task_url}.
  • Classic: every func is sent to the controller's /evaluate, one at a time, with its result and expected getters and its options. Results are combined by conj. No grading runs in the worker.
    • infeasible is the one exception: it is graded on the agent's answer (the [INFEASIBLE] tag), as cua_evaluate does.
    • The judge funcs get rubric and parameters, plus model=judge_model when the func's options set none. Deterministic funcs have any model option removed.
    • A per-func list (result, expected, options, rubric, parameters) whose length doesn't match the number of funcs is an error, not silently cut short.
  • Transport: /evaluate_async, then polling /evaluate_result. A controller without those routes (404) gets the blocking /evaluate. Every call carries the controller's bearer token.
  • Re-grades: stage_files (from step_overrides) is downloaded through /setup before anything reads the computer. The step refuses to grade if staging fails.
  • postconfig: runs once through /setup before grading. A failure is logged (error type plus at most 300 characters) and grading continues.
  • Skips: a turn whose error_type is set is recorded as a skip with score 0.
  • Partial credit: on classic tasks, an agent that ran out of turns or time is graded anyway and marked partial_credit (on the verification and in context.metadata["partial_credit"][verifier_id]). The check matches the stop message of every agent that ships (Max turns (N) reached, Reached maximum number of turns (N), and the agent, CLI, Letta Code, hermes and opencode timeouts).
  • Controller errors score 0 instead of failing the step.
  • Task refs: an http(s) URL is passed through. An object-store ref is signed for 6 hours, but only if it's in the object store and under <prefix>artifacts/; anything else is refused.

Result shape (same as cua_evaluate)

context.metadata["verifications"][verifier_id] = {"results": [...], "score": float}

  • V2: results=[{"id": "osworld_v2", "score", "result": not error, "message"}].
  • Classic: results=[{"func", "results": [{"id", "score", "result", "message"}]}], plus debug (overall_score, conjunction, metrics, error, message) and partial_credit when it applies.

Staging the app

python -m agent_env.task_step.task_steps.computer.osworld stage --source <OSWorld guest server source: main.py, pyxcursor.py> --out ./osworld-server
agent-env env put ./osworld-server

The guest server's source isn't vendored; it's staged under payload/. The app's declaration passes the checks a put runs. Then add the app to the computer env's apps. The app's requirements pin Pillow==13.0.0; every 12.x release has known vulnerabilities.

Dependencies

  • Controller (scaleapi/scale-cua#360, open): must

    • accept a full http(s):// REMOTE_VM_IP base URL that includes a path;
    • send REMOTE_SERVER_HEADERS on every call to the guest server;
    • require CONTROLLER_AUTH_TOKEN as a bearer token on every request.

    It must merge and its image must be published before this PR (see Merge order above).

  • collect_artifacts on computer envs: feat(collect_artifacts): collect from framework computer envs #601 (separate PR, open).

Known gaps

  • Formerly local funcs are now graded by the controller's OSWorld implementations, so scores may differ slightly from cua_evaluate's for the same task. These funcs are: compare_docx_files, compare_docx_tables, compare_pptx_files, compare_table, check_json_settings, check_json_keybindings, is_extension_installed, check_include_exclude, file_contains, run_sqlite3, check_thunderbird_prefs and check_thunderbird_filter.
    • Every one exists in the controller's metrics, and every getter type cua_evaluate reads (vm_file, cloud_file, rule, vm_command_line, the vscode getters) exists in its getters. So nothing becomes unservable.
  • The opencode judge's transcript isn't collected. It stays on the controller's sandbox.
  • Chrome DevTools getters (:9222), VLC (:8080) and the controller's WebSocket streams aren't reachable through the app. Only the guest server's HTTP routes are forwarded.
  • The accessibility tree relies on the OS's python3-pyatspi, which reaches the app's venv through sys.path. This needs checking on a live computer.
  • Setups that need root: the guest server runs as the session user.
  • App parity: the desktop apps OSWorld tasks expect (LibreOffice, GIMP, VLC, Thunderbird, VS Code) aren't in ubuntu-24.04.
  • Platform: Linux and X11 only.
  • End to end so far: local Docker only (see below). No agent, OSWorld-V2 .py task, LLM-judge func or cloud sandbox provider has run end to end.

Testing

  • Unit tests: 82 in tst/unit/task_step/osworld/ (guest resolution, controller transport, the three steps, the app). All pass.
    • Guest resolution: from the stored card, the live card read again, a computer missing the app, an env with no address, and deriving the controller token.
    • Controller transport: the bearer token on every call, HTTP errors with their status, non-JSON answers, /evaluate_async polling through transient errors, the 404 fallback, failed jobs, score normalization, signing refused outside the artifacts, require_auth refusing a controller that doesn't answer 401, and the token kept out of repr.
    • osworld_deploy_controller: env vars (including the token), metadata, the sandbox record, no secrets in the logs, refusal before any sandbox is created, no double deploy, a controller that doesn't require the token is refused, chain links each get their own addresses, prepare_image before create_container, a sandbox with a problem is let go, and every link failing.
    • osworld_initialize: V2 and classic, including site failures and error text cut to 300 characters.
    • osworld_evaluate: V2, a V2 controller error scoring 0, the error_type skip for V2 and classic, conj and/or across funcs, a single func's partial score, a failing func among others, infeasible, partial credit for each shipped agent's stop message (9 cases) and a message that isn't one, mismatched per-func list lengths, postconfig running once, and stage_files.
    • The app: the card extension, the forward (control token removed), 502 when the guest server is down, and staging.
  • Lint: ruff check passes.
  • CI: 34/34 checks passed before the rebase onto the latest main; rerunning on c3a2f60. The rebase's one conflict was rules.py, which keeps both main's browser steps and the three osworld_* steps.
  • Full local unit suite: 13,155 passed before the run was stopped. It hung on test_docker_image_github_clone (a credential test), and test_disk_recipe (Windows ISO boot) failed on macOS; neither touches this PR, and both pass in CI.

Docs

The site should cover:

  • the three steps and their fields;
  • the osworld-server app and the urn:agentenv:osworld-guest/v1 extension;
  • staging the guest server's source;
  • the controller image (controller_image_artifact_id) and its CONTROLLER_AUTH_TOKEN requirement.

End-to-end test

tst/integration/computer/osworld_test.py runs on local Docker:

  • a new computer env (osworld-desk: ubuntu-24.04, the reference driver, the osworld-server app);
  • the controller built from scale-cua#360 (with the token check) beside it;
  • a classic init_config execute set up through the controller;
  • then check_include_exclude over vm_command_line, graded by the controller. A rule that matches scores 1.0 and one that doesn't scores 0.0.

It passes against the token-checking controller image with Pillow 13 (132 s and 110 s on the latest two runs). It needs OSWORLD_SERVER_SOURCE and OSWORLD_CONTROLLER_IMAGE and skips without them (the osworld_sources capability).

The first live run found four bugs, fixed in a2b0fa7:

  • the controller got a loopback address its container can't reach;
  • the guest server's service command wasn't found;
  • python-xlib needed an ~/.Xauthority;
  • screenshots needed scrot.

The controller's address is also now saved in the run's context metadata, since a deployed env's record isn't re-saved after deploy.

Commits

  • cbe504c: the steps, the app and unit tests.
  • a2b0fa7: fixes from the first end-to-end run, and the integration test.
  • 7877592: the end-to-end test skips as the osworld_sources capability.
  • a443408: review fixes: the controller requires a token, each chain link gets its own addresses, StepContextError for context errors, bounded error text, and Pillow 13.
  • c3a2f60: prepare_image on each link, and every shipped agent's stop message.
  • 245ca40: the steps move to task_step/task_steps/computer/osworld/, beside snapshot_computer_env. No behavior change.
  • 20fdec6: Edgar's review: per-run judge key, endpoint and model; OSWorld's and/or; every agent_judge_* sent as a judge; a non-object answer scores 0; named timeouts; supervisord restarts the guest server.
  • ab7db64: type narrowing in osworld_evaluate.
  • 7ee469d: the rest of OSWorld (the controller client, the guest lookup, staging and its CLI, the osworld-server app) moves from agent_env.osworld into that same package, and the unit tests to tst/unit/task_step/osworld/. No behavior change; the end-to-end test passes (136 s).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The latest changes appear safe to merge; the previously reported wrong judge name remains a non-blocking issue.

Fix All in CursorFindings

  1. P2 Summary names the wrong judge ▶
Fix with agent prompt
### Issue 1
src/agent_env/task_step/task_steps/computer/osworld/evaluate.py:149-150
When a judge function sets `options["model"]` and the run has no `judge_model` override, this records the step's `judge_model` instead. `_payload` keeps the function's model, so the run summary names a model that did not grade the task. This makes score differences harder to explain.

Record the model selected for the request, including function-level options.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds OSWorld setup and grading on computer environments through a separate controller.

  • OSWorld tasks can reach a guest server through a separate controller beside the computer.
  • OSWorld tasks can set up a computer from classic instructions or a V2 task file.
  • OSWorld tasks can record classic or V2 evaluation results from the controller.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Worker[Task steps] -->|Bearer token| Controller[OSWorld controller]
  Controller -->|Computer control token| App[osworld-server app]
  App --> Guest[Local guest server]
  Controller -->|Scores| Worker
Loading

Reviews (14) · Last reviewed commit: "fix(osworld): narrow the prompt response..." · Reviewed by Greptile

@socket-security

socket-security Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​python3-xlib@​0.15991001007570
Addedpypi/​lxml@​6.1.39810010010080
Addedpypi/​pillow@​13.0.08710010010090
Addedpypi/​pyautogui@​0.9.54100100100100100

View full report

@socket-security

socket-security Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Priority Alert  (click "▶" to expand/collapse) Action
Low priority
Potential vulnerability: pypi pillow with risk level "medium"

Location: Package overview

From: src/agent_env/task_step/task_steps/computer/osworld/apps/osworld-server/requirements.txt → pypi/pillow@13.0.0

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/pillow@13.0.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn

View full report

@williamloo-scale williamloo-scale changed the title feat(task-steps): native OSWorld initialize and evaluate on computer envs feat(task-steps): native OSWorld initialize and evaluate on computer envs (FD-5492) Nov 11, 2026
@williamloo-scale
williamloo-scale marked this pull request as ready for review November 12, 2026 19:57
@williamloo-scale
williamloo-scale requested a review from a team as a code owner November 12, 2026 19:57
williamloo-scale and others added 5 commits November 12, 2026 15:45
…envs

Adds the osworld_deploy_controller, osworld_initialize and osworld_evaluate
steps and the osworld-server computer app (agent_env.osworld), so an OSWorld
task (classic init_config/evaluator or an OSWorld-V2 task_NNN.py) runs on a
framework computer env.

- The OSWorld controller runs in a container sandbox of its own beside the
  computer, in remote mode, so the grader, a task's checks and the judge's
  key stay out of the agent's reach (a computer's apps run as its session
  user).
- The computer runs only the OSWorld guest server, as the osworld-server
  app. Its card advertises urn:agentenv:osworld-guest/v1 at the prefix the
  app forwards to the guest server; the deploy step resolves the guest
  server's address from the computer's card (stored, else live until the
  app answers) and refuses a computer without the app before creating a
  sandbox.
- osworld_initialize: OSWorld-V2 via /reset + /initialize_task; classic via
  /setup with the whole init_config.
- osworld_evaluate: OSWorld-V2 via /evaluate; classic sends every func to
  the controller's /evaluate (async-polled, blocking fallback) and combines
  by conj; verifications shape matches cua_evaluate's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tion test (FD-5492)

A computer env with the osworld-server app, the controller beside it, a classic task set up and graded
through it: pass scores 1.0, fail 0.0. Running it found:

- The controller got the guest server's address as this process reaches it; a local container reaches
  this machine's loopback by another name. It now gets it, and the model endpoint, through
  url_from_sandbox, as an agent's deploy does.
- supervisord looks a bare command up on PATH, so the guest server's service never started: it runs
  as `bash start-osworld-server.sh`.
- python-xlib refuses to connect without an Xauthority file; the start script makes an empty one.
- pyautogui's screenshots need scrot on X11; the app installs it.
- A deployed env's record is stored once, when it is deployed, so the controller's address written on
  it lasted only the process. It is recorded in the run's context metadata too, and read from there.

tst/integration/computer/osworld_test.py stages the app from OSWORLD_SERVER_SOURCE and runs the controller
image OSWORLD_CONTROLLER_IMAGE (one with scale-cua #360); it skips without them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ility (FD-5492)

It needs scale-cua's guest server source and a controller image, which this repository doesn't carry,
so it skips without them. CI's skip policy fails a skip that isn't a declared capability gap, so the
skip names osworld_sources, the slow tier allows it as it does mcp_server_sources, and CONTRIBUTING
asks a change to the OSWorld code to run the test with both set and say so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… addresses, and review fixes (FD-5492)

- osworld_deploy_controller starts the controller with CONTROLLER_AUTH_TOKEN, an HMAC of the env's
  control token, and refuses one that answers a call without it: its port can be reachable from the
  computer, and so by the agent. Every controller call carries the token (controller.Endpoint).
- The controller's sandbox is created link by link, each given the guest server and model endpoint
  as its own sandboxes reach them; a chain's url_from_sandbox can't say for the link it ends up on.
- osworld_evaluate refuses a result/expected/options list whose length isn't the funcs'.
- A failed setup is reported by its error alone, bounded, never the controller's whole answer.
- Context failures raise StepContextError; the controller's disk is a named constant.
- The osworld-server app pins Pillow 13.0.0, past every Pillow advisory.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ry shipped agent's budget stop (FD-5492)

- osworld_deploy_controller calls the link's prepare_image before creating its container, so an image
  stored as its build context alone is built rather than pulled, and lets go a sandbox the link says
  can't run the image for the next link's, as the provider chain does.
- osworld_evaluate reads Claude Code's and Codex's "Reached maximum number of turns (N)" and each CLI's
  "<name> timed out after Ns" as a budget stop, grading the desktop and marking partial credit, beside
  the harness's own sentences. A stall, a tool's timeout or any other error still isn't one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale
williamloo-scale force-pushed the wloo/osworld-native-computer-steps branch from b1e420c to c3a2f60 Compare November 12, 2026 23:47
williamloo-scale and others added 2 commits November 12, 2026 17:23
…ter (FD-5492)

osworld_deploy_controller, osworld_initialize and osworld_evaluate run on computer envs, so they
live in task_steps/computer/osworld beside snapshot_computer_env. No behavior change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…world (FD-5492)

The controller client, the guest-server lookup, staging and its CLI, and the osworld-server
computer app move from agent_env.osworld into the steps' own package,
agent_env.task_step.task_steps.computer.osworld, so OSWorld lives in one place beside the other
computer steps. The unit tests move to tst/unit/task_step/osworld. No behavior change.

Stage the app with: python -m agent_env.task_step.task_steps.computer.osworld stage ...

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@earakely-scale earakely-scale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 7ee469d. The two commits since c3a2f60 only move files and change import paths. Comment 1 is the question I'd settle first; 2 to 4 are correctness; 5 collects small things.

logger = logging.getLogger(__name__)

#: The controller image's artifact id when a step names none.
DEFAULT_CONTROLLER_IMAGE_ARTIFACT_ID = "cua-controller"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question to settle before the rest: should this be in core?

The controller this step deploys is built from scale-cua, which is private. This default names an image in Scale's store, and the env the controller gets (PROVIDER=remote, REMOTE_VM_IP, REMOTE_SERVER_HEADERS, CONTROLLER_AUTH_TOKEN) is scale-cua's contract. Upstream OSWorld has no HTTP controller, no remote provider and no /evaluate_async. So nobody outside Scale can run these three steps, and the end-to-end test can never run in public CI: it adds an osworld_sources skip to the slow tier instead. AGENTS.md asks core to stay vendor-neutral and not to add a seam that nothing in the repository consumes or defaults. Moving the package under task_steps/computer puts it further into core.

Two ways out:

  • A plugin: the steps through [task_steps] impls and the app as its own env, from scale-cua or agentenvhub-sdk. Every piece already has an extension point.
  • Public pieces: the guest server (main.py, pyxcursor.py) is upstream OSWorld's, Apache-2.0, so it could be a pinned download checked by sha256, as the drivers' installs are. The controller would need a public image built in this repository, so CI runs the test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that as written nobody outside Scale can run these steps: the controller image comes from private scale-cua, and the env it gets (PROVIDER=remote, REMOTE_VM_IP, REMOTE_SERVER_HEADERS, CONTROLLER_AUTH_TOKEN) is scale-cua's contract. We'll take both of your ways out, in parallel:

  1. Now, Scale-only: a plugin. The three steps, the controller client, the guest lookup and the osworld-server app go into agentenvhub-sdk as agent_env.task_steps entry points, next to the cua_* steps the hub already runs from there. That unblocks the hub (scaleapi/scaleapi#174344) without anything vendor-specific in core.

  2. In parallel, core: public pieces. This PR stays open as the core version and doesn't merge until the steps run on public parts:

    • Guest server: upstream OSWorld's main.py and pyxcursor.py (Apache-2.0) as a pinned download checked by sha256, like the drivers' installs, instead of staging scale-cua's copy.
    • Controller: an image built in this repository from public code, so the end-to-end test runs in public CI and the osworld_sources skip goes away.
    • Default: controller_image_artifact_id defaults to that public image, not a Scale store id.
    • Contract: the env the step sets becomes the public controller's documented contract.

    Once core has these, the plugin's copies go away and the hub uses the core steps.

Comments 2 to 5 are fixed here (20fdec6), so both versions start from the same code. On staging, from comment 5: in core the app becomes a bundle folder beside the other shipped apps, with an agent-env subcommand, as part of step 2.

CONTROLLER_TOKEN_ENV: guest.controller_token(deployed)}
config = get_config()
try:
env["LITELLM_API_KEY"] = config.get_litellm_api_key()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per-run judge overrides don't reach the controller. This passes the configured [model] key only, so a run's user_overrides.judge_litellm_api_key (or litellm_api_key) is ignored. osworld_evaluate reads only its own judge_model field, not user_overrides.judge_model. rubrics_verifier honours all three (rubrics_verifier.py:772 and 814), and AGENTS.md makes user_overrides the contract for per-run keys.

So agent-env task run --judge-litellm-api-key K --judge-model M grades the judge funcs with another key and the controller's default model. With no [model] key configured, the judge gets no key, and those funcs score 0.

Suggest resolving the key as rubrics_verifier does, config.resolve_reference(overrides.get("judge_litellm_api_key") or overrides.get("litellm_api_key")), then falling back to [model]. And let user_overrides.judge_model set the judge funcs' model ahead of the step's own judge_model, as rubrics_verifier does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 20fdec6, following rubrics_verifier:

  • Key: osworld_deploy_controller passes config.resolve_reference(user_overrides.judge_litellm_api_key or user_overrides.litellm_api_key), and falls back to [model]'s key when the override is unset or resolves to nothing.
  • Base URL: judge_litellm_base_url / litellm_base_url work the same way.
  • Model: osworld_evaluate sends user_overrides.judge_model as the judge funcs' model. It wins over a func's own model option and over the step's judge_model, and it's recorded with context.record_model("judge", ...).

Tests cover the judge key, the fallback to the run's key and then to [model], and the model override.

if len(scores) == 1:
score = scores[0]
else:
score = aggregate_score([{"score": s, "result": s == PASS_SCORE} for s in scores],

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several funcs are combined as pass/fail, which isn't how OSWorld combines them. A func counts as passed only at exactly 1.0, and ALL_PASS/ANY_PASS return 1 or 0, while a single func keeps its partial score.

OSWorld's DesktopEnv.evaluate (desktop_env.py L580-587):

  • and returns 0 when any metric is 0, else their mean;
  • or returns 1 when any metric is 1, else their max.

So and over [1.0, 0.5] scores 0.0 here and 0.75 in OSWorld, and or over [0.5, 0.3] scores 0.0 here and 0.5 in OSWorld. If this deliberately matches cua_evaluate, say so in the docstring. Otherwise suggest following OSWorld's rule, since these steps exist to give OSWorld's scores.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 20fdec6. Several funcs now combine as OSWorld's DesktopEnv.evaluate does: and is 0 when any func scores 0, otherwise their mean; or is their max. That's also what the controller does itself (scale_cua_environment.py:871-880), so a task scores the same either way. One func keeps its own score, as before.

The conj test has your two cases: and over [1.0, 0.5] scores 0.75, and or over [0.5, 0.3] scores 0.5.


def __init__(self, id: str, version: Optional[int], env_id: str, evaluator: Optional[dict] = None,
verifier_id: Optional[str] = None, osworld_v2_task_path: Optional[str] = None,
osworld_v2_task_url: Optional[str] = None, judge_model: Optional[str] = None,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Smaller ones, none blocking:

  • osworld_v2_task_url is called the older name, here and in initialize.py:26, but these step types are new, so no stored task has it. Unless hub tasks are being converted into these types, suggest dropping it (AGENTS.md: no defensive code for a case no caller produces).
  • _BUDGET_EXHAUSTED (line 37) matches each shipped agent's stop sentence as free text, so a reworded message silently turns partial credit into a skip. A structured stop reason on PromptResponse would hold up better. Short of that, a test could check each pattern against the agent's own source.
  • start-osworld-server.sh restarts the server in its own loop (line 9), under a service manager that already restarts services. One restart policy is enough.
  • The timeouts at controller.py:84 (60) and controller.py:131 and 146 (15) want names, as Greptile noted.
  • Staging is python -m agent_env.task_step.task_steps.computer.osworld stage ... since the move. An agent-env subcommand, or a bundle folder like browsers/envs/chromium, would be easier to find, and would put the app where the other shipped apps live.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All in 20fdec6 unless noted:

  • osworld_v2_task_url: kept, because hub tasks are being converted into these types. scaleapi/scaleapi#174344 makes the hub's CUA importer emit osworld_initialize / osworld_evaluate with osworld_v2_task_url when a CSV row gives a task URL, as it does for the cua_* steps. The docstrings now say that instead of "older name".
  • _BUDGET_EXHAUSTED: added test_each_agents_budget_stop_is_read_as_its_source_writes_it. For each in-repo bridge (codex, claude-code, gemini, hermes, letta-code, opencode), it checks that the bridge's source still contains its stop f-string and that the pattern matches that sentence, so a rewording fails the test. A structured stop reason on PromptResponse would hold up better; it touches every bridge, so I'd rather do it as its own change.
  • start-osworld-server.sh: the loop is gone. It execs the server, and the composer's supervisord program (autorestart=true, composer.py:960) restarts it. The end-to-end test passes with this.
  • Timeouts: named START_TIMEOUT_S (60) and READY_CALL_TIMEOUT_S (15).
  • Staging: this depends on where the code ends up after your first comment, so I'll settle it there.

…object answers (FD-5492)

- osworld_deploy_controller gives the judge the run's key and endpoint
  (user_overrides.judge_litellm_api_key / litellm_api_key, judge_litellm_base_url /
  litellm_base_url) ahead of [model]'s, as rubrics_verifier does.
- osworld_evaluate gives judge funcs user_overrides.judge_model ahead of their own option and
  the step's judge_model, records it as the run's judge model, and treats every agent_judge_*
  variant as a judge (agent_judge_zip, _blend, _godot, _log, _gz, _fcstd were sent as
  deterministic funcs, without the model or the rubric).
- Several funcs combine as OSWorld's DesktopEnv.evaluate and the controller do: `and` is 0
  when any func scores 0, else their mean; `or` is their max.
- A controller answer that isn't a JSON object raises ControllerCallError, which the steps
  score 0, instead of an AttributeError that failed the step.
- The controller timeouts are named; start-osworld-server.sh leaves restarts to supervisord;
  the osworld_v2_task_url field is documented as what the hub's importer writes.
- A test pins each shipped agent's budget stop sentence to its bridge's source.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment on lines +149 to +150
if any(is_judge(func) for func, *_ in self._funcs()):
context.record_model("judge", self.id, run_judge_model or self.judge_model)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Summary names the wrong judge

When a judge function sets options["model"] and the run has no judge_model override, this records the step's judge_model instead. _payload keeps the function's model, so the run summary names a model that did not grade the task. This makes score differences harder to explain.

Record the model selected for the request, including function-level options.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/computer/osworld/evaluate.py
Line: 149-150

Comment:
**Summary names the wrong judge**

When a judge function sets `options["model"]` and the run has no `judge_model` override, this records the step's `judge_model` instead. `_payload` keeps the function's model, so the run summary names a model that did not grade the task. This makes score differences harder to explain.

Record the model selected for the request, including function-level options.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

…e used (FD-5492)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants