Repository navigation
Add Databricks sandbox stdio executor (first-class agents.sandbox API + native stdio:// admission) - #3
Open
HsingTang wants to merge 9 commits into
Open
Add Databricks sandbox stdio executor (first-class agents.sandbox API + native stdio:// admission)#3HsingTang wants to merge 9 commits into
HsingTang wants to merge 9 commits into
Conversation
HsingTang
force-pushed
the
hsingtang/databricks-codex
branch
2 times, most recently
from
November 11, 2026 17:36
0adf796 to
7011b5e
Compare
HsingTang
force-pushed
the
hsingtang/databricks-codex
branch
10 times, most recently
from
November 16, 2026 03:48
febe920 to
34d868c
Compare
Add examples/codex/databricks: an application-owned gateway and a lifetime-managing Environment context manager that back the native executor with a Databricks sandbox, following the remote-execution pattern in docs/sdk/execution.md (the Databricks counterpart of offline_docker.py). The gateway relays the SDK's WSS executor connection to the sandbox exec-server over the databricks-sdk streaming command API; databricks_sandbox_environment starts the gateway and yields a configured Environment. Ship as an example (not a src/agents module or dependency extra) to match how the branch ships executor gateways. Include a runnable runner, a README, example requirements, and tests for the relay, bearer auth, teardown paths, the context manager, and the SSL_CERT_FILE trust helper. The gateway drops the non-JSON startup banner the sandbox's codex prints on stdout before the exec-server protocol, so it cannot corrupt the stream; a live turn confirmed the banner is dropped and the tool still executes. The runner and the context manager confirm the sandbox is running (starting it if stopped) before launching the exec-server, since execute_command needs a running sandbox. Validated end-to-end against a live sandbox: the exec-server stdio mode, bidirectional streaming over execute_command, and newline framing, plus exec_command / write_stdin / apply_patch, multi-turn with session resume and a real sandbox stop-start, large and single-line output, and the Environment approval / transform_arguments / transform_result hooks with an enforced NetworkPolicy. tests/codex pass (14) and mypy --strict is clean against the preview wheel on a glibc-2.39 host. This also fixes the runner's sandbox-image spec, which set a non-existent SandboxSpec.image instead of SandboxSpec.environment.image_uri, and caps inbound WebSocket frames at 4 MiB so a large apply_patch/write_stdin payload is not dropped. The runner defaults the model provider to the sandbox workspace's Databricks AI Gateway, with a token minted from the Databricks profile (PAT or OAuth), so no OpenAI key is needed; set OPENAI_BASE_URL / OPENAI_API_KEY to override. It accepts the sandbox id or the sandboxes/<id> resource name and creates the workspace directory before the turn. Both model-provider paths - the AI Gateway default and the OpenAI-direct override - are validated end to end. Co-authored-by: Isaac <no-reply@databricks.com>
HsingTang
force-pushed
the
hsingtang/databricks-codex
branch
from
November 16, 2026 04:58
34d868c to
e90be32
Compare
Replace the wss gateway example with a DatabricksStdioEnvironment that subclasses the native Environment and declares a host-owned stdio_command, mirroring the SDK's Docker stdio executor. The native runtime spawns a local launcher (launch.py) and speaks the exec-server protocol over its stdio; the launcher relays that to `codex exec-server --listen stdio` in the sandbox over the databricks-sdk streaming command API. This drops the loopback WebSocket server, self-signed TLS, CA-trust juggling, and bearer-token handshake. Relax the native stdio admission guard to accept a sanctioned stdio:// host-owned source alongside docker://, so non-Docker backends can supply a trusted host-owned stdio executor; the headers-empty invariant is unchanged. Add Rust unit tests for stdio:// acceptance and unknown-scheme rejection. Rewrite the example tests to cover stdio_command emission, stdio:///header validation, the factory precondition and credential/launcher wiring, and the launcher's startup-banner filter. Co-authored-by: Isaac <no-reply@databricks.com>
…box API Move the Databricks stdio executor from example code into the SDK package as a supported, exported class: agents.sandbox.DatabricksStdioEnvironment, beside DockerGateway. It subclasses the native Environment, emits a host-owned stdio_command, and provides a for_sandbox(workspace_client, sandbox_name, cwd) constructor that confirms the sandbox is running and derives the host/token the launcher authenticates with. The launcher (_databricks_launcher.py) bridges the runtime's child stdio to codex exec-server in the sandbox over databricks-sdk; databricks-sdk is imported lazily so importing the module does not require it. The example now consumes the first-class class (runner uses DatabricksStdioEnvironment.for_sandbox); its environment.py/launch.py are removed. Tests move to the public path (agents.sandbox) and cover stdio_command emission, stdio://-url/header validation, the for_sandbox precondition and wiring, and the launcher banner filter. Note: agents.sandbox pulls agents.classic, so importing the class requires the [classic] extra. released_api_contract.json must be regenerated for the new public export during the native build (make update-released-api-contract); it is not editable without that build. Co-authored-by: Isaac <no-reply@databricks.com>
- Launch the bridge as a module (python -m _agents_codex._databricks_launcher) instead of exec'ing a file path, so it resolves via the module system. - Harden the launcher: forward the exec-server's stderr for diagnostics, surface the remote command's exit code instead of always returning 0, and close the exec-server's stdin on teardown/SIGTERM/SIGINT so it shuts down (the SDK cannot kill the remote command; a lingering one is reaped by the sandbox inactivity timeout). Factor the relay into a testable _run. - Move the tests into packages/codex/tests so the default testpaths (and CI) collect them, and add coverage for the launcher's credential guard, protocol forwarding, stdin close, and exit-code handling. - Note that for_sandbox freezes the derived token, so a fresh environment is needed per session with short-lived OAuth/U2M credentials. Co-authored-by: Isaac <no-reply@databricks.com>
- Set the done event in the SIGTERM/SIGINT handler so the launcher's main loop always returns, even if closing the stream does not unblock the relay read (previously it could ignore SIGTERM until SIGKILL). Factor the teardown into a module-level _terminate for testing. - Make the environment tests run without databricks-sdk / the [classic] extra: import the class from its owning module and satisfy for_sandbox's lazy databricks.sdk import with an injected fake. Add missing-token and None-status cases and an agents.sandbox export assertion. - Cover the launcher's stdin relay (and close-on-write-error), the terminate / signal-installation path, and main()'s execute_command call shape. Co-authored-by: Isaac <no-reply@databricks.com>
Correct the Tests-section claim (the tests run without databricks-sdk / the [classic] extra via an injected fake), lead with the first-class class, name the [classic] extra (openai-agents[classic]) needed by the public import path, drop the invented "v1.0" label, add stderr to the CommandProcess description, and fix tests/codex -> packages/codex/tests. Co-authored-by: Isaac <no-reply@databricks.com>
Cut the Status, How it works, Files, and Tests sections; add a Usage example Python snippet; rename the run section to "Install and run the example" and make the shell blocks clearly launch runner.py; use my-sandbox in all examples; call the config a .databrickscfg profile (not a CLI profile); qualify DatabricksStdioEnvironment.for_sandbox in prose; rework the codex-binary prerequisite; and add a References section. Co-authored-by: Isaac <no-reply@databricks.com>
Drop the redundant "only the model provider changes" line and the stdio:// SDK-build prerequisite, and rewrap the credentials paragraph. Co-authored-by: Isaac <no-reply@databricks.com>
Resolve OPENAI_BASE_URL and OPENAI_API_KEY as a pair in runner.py: with neither set, use the Databricks AI Gateway and a profile token; with both, use the given provider; with exactly one, raise instead of silently sending a key to the wrong endpoint. Document the pairing in the README, and correct the SANDBOX_CREATE docstring - the runner creates unconditionally when set, it does not check for an existing sandbox. Co-authored-by: Isaac <no-reply@databricks.com>
HsingTang
marked this pull request as ready for review
November 17, 2026 04:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Databricks Sandbox now supports OpenAI Agents SDK
feat/codex-harnessintegration: an agent driven by this SDK runs its shell and file tools (exec_command/write_stdin/apply_patch) inside a Databricks sandbox. Opened in a personal fork (basefeat/codex-harness, headhsingtang/databricks-codex) for the Databricks and OpenAI teams to review.How it works
Your application runs the agent loop and connects to a codex process in the sandbox - the same shape as the SDK's own Docker support, offered as a first-class class:
agents.sandbox.DatabricksStdioEnvironment, besideDockerGateway. Build it withDatabricksStdioEnvironment.for_sandbox(workspace_client, sandbox_name, cwd), which confirms the sandbox is running and derives the host and token the launcher authenticates with. When the session opens, the Agents SDK starts a launcher that runs codex exec-server in the sandbox and relays it over thedatabricks-sdkcommand API (databricks-sdkis imported lazily).Because the runtime owns the launcher over ordinary pipes, there is no gateway server, no TLS, and no inbound ingress; the sandbox is reached outbound through the Databricks control plane. The agent loop runs in your process, so the
Environmentapproval / transform /NetworkPolicyhooks stay available, and the model runs whereverNativeProviderpoints - the workspace's Databricks AI Gateway (no OpenAI key) or OpenAI directly.Native Agents SDK changes for review
Today the native runtime admits a host-owned
stdio_commandonly when the executor url starts withdocker://("stdio executor requires a current SDK Docker source"). So a non-Docker backend can run its own executor,packages/codex/rust/environment.rsnow also accepts a sanctionedstdio://source, with the headers-empty invariant unchanged and Rust unit tests for acceptance and unknown-scheme rejection.Open questions for OpenAI reviewers:
stdio://the right shape for a sanctioned non-Docker host-owned stdio executor?Environment._to_native/_validate_remote_url; happy to switch to a public constructor if you'd rather not expose those as extension points.Validation
End-to-end against a live Databricks sandbox on the released
0.24.0-alpha.8wheel: a real codex exec-server ran the agent's tool call in the sandbox with no gateway and no TLS, overdatabricks-sdk, for both model providers (AI Gatewaygpt-5.5and OpenAI-directgpt-5.3-codex). The Python layer -stdio_commandemission,stdio://and header validation, thefor_sandboxprecondition and wiring, and the launcher's banner filter - passes inpackages/codex/tests/through the publicagents.sandboxpath. Thestdio://admission is covered by Rust unit tests. The auth path is exercised with both PAT and OAuth U2M, confirmed against a live sandbox (workspace auth +execute_command).