Repository navigation
feat(drivers): codex-computer-use, OpenAI's Codex Computer Use as a macOS driver (FD-5356) #550
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
earakely-scale
merged 5 commits into
main
from
edgararakelyan/fd-5356-cu2-codex-computer-use-driver
Nov 9, 2026
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
5390455
feat(drivers): codex-computer-use, OpenAI's Codex Computer Use as a m…
earakely-scale 1c900da
fix(drivers): close codex-computer-use's idle REPLs, and don't call j…
earakely-scale 3e14f51
fix(drivers): calls a session makes together share one codex-computer…
earakely-scale 37ed4de
refactor(drivers): codex-computer-use reads approvals from their meta…
earakely-scale cfca9f6
fix(drivers): a codex-computer-use call right after its REPL stops st…
earakely-scale File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
352 changes: 352 additions & 0 deletions
352
src/agent_env/drivers/envs/codex-computer-use/driver.py
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| type = "computer_driver" | ||
| module = "driver.py:CodexComputerUse" | ||
| port = 18765 | ||
|
|
||
| [install.macos] | ||
| python = "requirements-macos.txt" | ||
| downloads = [ | ||
| { url = "https://persistent.oaistatic.com/codex-app-prod/ChatGPT-darwin-arm64-26.930.51102.zip", sha256 = "7001df293fd478b2a4e2d8b5f4ea3aae43a2e21d168960256b2f31ac4ee43f6f", to = "ChatGPT.zip", arch = "arm64" }, | ||
| ] | ||
| run = [{ script = "install.sh", scope = "session" }] | ||
| grants = [ | ||
| { executable = "venv/bin/python", services = ["accessibility", "post_event", "screen_capture"] }, | ||
| { app = "/opt/agentenv/codex-computer-use/Codex Computer Use.app", services = ["accessibility", "post_event", "screen_capture"] }, | ||
| ] | ||
| services = [{ name = "driver", command = "venv/bin/python -m agentenv_protocol.computer.serve driver.py:CodexComputerUse" }] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| #!/bin/sh | ||
| # The pinned ChatGPT app, from the zip the install downloaded and checked, unpacked beside the driver and never run, so it | ||
| # never updates itself; and Codex Computer Use's helper app copied out of it to where the layer's grant names it, a path | ||
| # that doesn't depend on this folder's name. As the session user, whom the helper must belong to. | ||
| set -eu | ||
| ditto -x -k ChatGPT.zip . | ||
| rm -f ChatGPT.zip | ||
| helper="Codex Computer Use.app" | ||
| dest=/opt/agentenv/codex-computer-use | ||
| mkdir -p "$dest" | ||
| rm -rf "$dest/$helper" | ||
| ditto "ChatGPT.app/Contents/Resources/cua_node/lib/node_modules/@oai/sky/$helper" "$dest/$helper" | ||
| xattr -dr com.apple.quarantine ChatGPT.app "$dest" || true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| { | ||
| "instructions": "UI automation through cua_repl using the initialized cua API.", | ||
| "tools": [ | ||
| { | ||
| "name": "js", | ||
| "description": "Control native apps or browsers on the user’s computer by reading or operating UI. Prefer purpose-built skills, connectors, APIs, or CLIs when available.\n\nOn the first invocation of `cua_repl`, or after resetting it, execute exactly one of the API calls shown below, optionally assigning its result to a variable. Do not add other API calls, waits, or snapshots to that invocation.\nThe tool result will include documentation and, when creating or selecting a tab or selecting an app, its initial UI state. Selecting a browser does not open a tab. Read that result before continuing.\nUse only APIs described in the tool instructions or returned documentation.\n\nWhen you need an inventory of available apps, browsers, and tabs, get a snapshot of all enabled surfaces. Otherwise, use the relevant entry point below:\n\n```javascript\nawait cua.getState();\n```\n\nBrowser APIs are disabled.\n\nIf the user specifies an app to use, get the app by name, bundle ID, or path:\n\n```javascript\nlet app = await cua.getApp(\"Example App\");\n```\n\nTo add other content to the tool result, use `nodeRepl.write(value)` for text or other values and `await nodeRepl.emitImage(image)` for images. The APIs listed above already display their documentation or UI state; do not wrap their results in `write` or `emitImage`.\n\nIf your context begins with a summary of an existing computer use task, call `await cua.rewriteDocumentation()` before continuing the computer use task to ensure you have a complete view of the necessary documentation.", | ||
| "inputSchema": { | ||
| "additionalProperties": false, | ||
| "properties": { | ||
| "code": { | ||
| "description": "JavaScript to execute using the initialized cua_repl runtime.", | ||
| "type": "string" | ||
| }, | ||
| "timeout_ms": { | ||
| "description": "Optional execution timeout in milliseconds. Defaults to 30000 (30 seconds) when omitted.", | ||
| "minimum": 1, | ||
| "type": "integer" | ||
| }, | ||
| "title": { | ||
| "description": "Short user-facing description of what the code does.", | ||
| "maxLength": 80, | ||
| "minLength": 1, | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "code" | ||
| ], | ||
| "type": "object" | ||
| }, | ||
| "annotations": { | ||
| "readOnlyHint": true, | ||
| "destructiveHint": false, | ||
| "openWorldHint": false | ||
| } | ||
| }, | ||
| { | ||
| "name": "js_add_node_module_dir", | ||
| "description": "Add an absolute `node_modules` directory for package imports. The directory remains available after `js_reset`.", | ||
| "inputSchema": { | ||
| "additionalProperties": false, | ||
| "properties": { | ||
| "path": { | ||
| "description": "Absolute path to a node_modules directory to add to Node package resolution.", | ||
| "minLength": 1, | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "path" | ||
| ], | ||
| "type": "object" | ||
| }, | ||
| "annotations": { | ||
| "readOnlyHint": true, | ||
| "destructiveHint": false, | ||
| "openWorldHint": false | ||
| } | ||
| }, | ||
| { | ||
| "name": "js_reset", | ||
| "description": "Reset the persistent cua_repl JavaScript session. All JavaScript bindings are discarded. The next cua_repl.js call initializes a fresh runtime for the enabled surfaces. This does not close browser tabs or native apps, or erase their state.", | ||
| "inputSchema": { | ||
| "additionalProperties": false, | ||
| "properties": {}, | ||
| "type": "object" | ||
| }, | ||
| "annotations": { | ||
| "readOnlyHint": true, | ||
| "destructiveHint": false, | ||
| "openWorldHint": false | ||
| } | ||
| }, | ||
| { | ||
| "name": "turn_ended", | ||
| "description": "Notify trusted libraries that a Codex turn ended. Repeated notifications for the same session and turn are ignored.", | ||
| "inputSchema": { | ||
| "additionalProperties": false, | ||
| "properties": { | ||
| "hook_event_name": { | ||
| "minLength": 1, | ||
| "type": "string" | ||
| }, | ||
| "session_id": { | ||
| "minLength": 1, | ||
| "type": "string" | ||
| }, | ||
| "turn_id": { | ||
| "minLength": 1, | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "hook_event_name", | ||
| "session_id", | ||
| "turn_id" | ||
| ], | ||
| "type": "object" | ||
| }, | ||
| "annotations": { | ||
| "idempotentHint": true | ||
| } | ||
| } | ||
| ] | ||
| } | ||
4 changes: 4 additions & 0 deletions
4
src/agent_env/drivers/envs/codex-computer-use/requirements-macos.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| # What a composed macOS computer installs into the driver's venv: the protocol with MacOSDriver, whose screen and input | ||
| # need its macos extra, and the MCP client the forwarded tools reach Codex's cua_repl server with. A release pins it to | ||
| # the version it ships. The ChatGPT app is the pinned download, whose executables' sha256 the driver checks. | ||
| agentenv-framework-protocol[computer,macos]>=0.1.459 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.