Repository navigation
Add internal sandbox WebSocket SSH server - #7588
Conversation
✅ Integration test reportCommit: cd5c216
Top 3 slowest tests (at least 2 minutes):
|
d3e8539 to
e27bc56
Compare
e27bc56 to
36dc63c
Compare
rclarey
left a comment
There was a problem hiding this comment.
The relay looks careful overall. Admission and the idle decision share a lock, readiness is announced only after sshd -t passes and the listener is bound, ReadHeaderTimeout is set, and shutdown waits for in-flight handlers. The ConnectionsManager change also looks safe for compute, because waitForIdleShutdown rechecks Count(). I ran the Server tests 5 times with -race and they all passed.
Inline comments cover the specific issues. Two broader points, compared with ssh connect:
- Resume and handover. These are the main reliability features of the compute tunnel, and this relay rejects both. The README in #7590 acknowledges that a broken connection ends the session. Does sandbox port forwarding enforce a maximum WebSocket lifetime, like the driver proxy? If it does, long sessions will drop on a schedule. Handover exists in compute to work around exactly that.
- Logging. The compute server runs with
--log-level=info --log-file=stdoutand logs lifecycle events such as idle shutdown.Servelogs nothing: not readiness, not 503 rejections, not idle shutdown. #7589 doesn't pass a log level either, so only warnings reach the client's diagnostics stream. That leaves nothing to look at when a connection fails to come up. It would help to log the lifecycle events at info level and have #7589 pass--log-level.
anton-107
left a comment
There was a problem hiding this comment.
Requesting changes for code ownership of experimental/ssh/sandbox/, see the inline comment. It's the CODEOWNERS side of rclarey's depguard comment on .golangci.yaml:60.
36dc63c to
cb5c0cf
Compare
|
@rclarey Addressed the two broader review points as well. The sandbox client now renews the WebSocket every 25 minutes using same-session handover; unexpected-disconnect resume remains disabled. A local reverse-proxy test enforces a short WebSocket lifetime and verifies that repeated handovers keep the same SSH process and data stream alive. The server now logs readiness, rejected connections, first-connection timeout, and shutdown at info level. The client starts it with --log-level info and sends diagnostics to stderr, keeping stdout available for SSH traffic. These changes are in cb5c0cf and 637d42e. Local tests and linters pass; live workspace validation is still pending. |
cb5c0cf to
a6b20fd
Compare
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 1 tests: regression=1 unreleased=0 coverage=0 failing=0
TestAccept/cmd/sandbox/server: branch ✅ | main (06e25a1) ❌ | latest (v1.27.0) ❌main (06e25a1): latest (v1.27.0): same output as main (06e25a1) |
a6b20fd to
219d06c
Compare
| closing := false | ||
| handler := proxy.NewProxyServer(ctx, connections, func(ctx context.Context) *exec.Cmd { | ||
| mu.Lock() | ||
| close(started) |
There was a problem hiding this comment.
After a failed sshd start the proxy frees the session for a retry, but this factory closes started unconditionally, so a second dial that lands before Serve sets closing panics here with mu held, and Serve can then hang in its deferred mu.Lock() with SIGTERM already captured by NotifyContext. Our client only dials one new session per server so it's unlikely in practice, but guarding the close with a sync.Once (or keeping the session consumed on a start failure, since Serve exits on it anyway) would close it.
There was a problem hiding this comment.
Fixed by keeping single-session admission consumed once the process factory is invoked. Only a failed WebSocket upgrade releases it. An sshd startup error still reaches Serve, but another request cannot invoke the factory and close started again. Tests cover stdin/stdout/Start failures, rejection of a second session, and 16 concurrent startup attempts; repeated race checks pass.
anton-107
left a comment
There was a problem hiding this comment.
Requesting changes for one issue, see the inline comment on server.go:64. It's the double close from rclarey's round-2 comment, which I reproduced. Everything else from the earlier rounds looks addressed, and the CODEOWNERS entry resolves my previous request.
| closing := false | ||
| handler := proxy.NewProxyServer(ctx, connections, func(ctx context.Context) *exec.Cmd { | ||
| mu.Lock() | ||
| close(started) |
There was a problem hiding this comment.
Blocking: this panics on a retry after a failed sshd start. It's the same issue as rclarey's round-2 comment above, which is still open.
WithSingleSession sets sessionUsed back to false when the process didn't start, so a second new-id dial can reach this factory again before Serve's deferred closing = true runs. The second close(started) then panics with mu held. net/http recovers the panic, but mu stays locked. If that happens before Serve's defer calls mu.Lock(), Serve deadlocks, and SIGTERM has already been consumed by NotifyContext.
Repro: a fake sshd that passes -t and then deletes itself (rm -f "$0"), so sshd -i fails Start with ENOENT. Then 16 goroutines keep dialing /ssh?id=s<n> with the token. In 6 of 20 runs the log shows http: panic serving ...: close of closed channel. I didn't catch the deadlock itself; in those runs the panic came after the defer had released mu.
The client dials once, so this is unlikely in practice, but the fix is small. I'd keep the session consumed once the factory has run, instead of reopening it in handleNewConnection, because Serve exits on a start failure anyway. That also removes the processStarted reset branch. A sync.Once around the close would work too.
There was a problem hiding this comment.
Applied the admission change you suggested: invoking the command factory consumes the single session even if sshd fails to start, while failed WebSocket upgrades remain retryable. Removed the processStarted reset logic. The proxy regression asserts exactly one factory invocation and completion under 16 concurrent dials; the sandbox test also makes 16 concurrent attempts after removing sshd and verifies that Serve returns. Race tests and the full SSH suite pass.
anton-107
left a comment
There was a problem hiding this comment.
The double close on server.go:64 is fixed in 605f0a0. Once the process factory runs, the session stays used, so a second dial can't reach close(started) again. I reran the repro (fake sshd that deletes itself, 16 goroutines dialing in a loop) and got 0 panics in 40 runs, down from 6 in 20. The sandbox and proxy packages also pass with -race (count=3). LGTM.
…atabricks#7589) ## 🥞 Stacked PR Use this [link](https://github.com/databricks/cli/pull/7589/files) to review incremental changes. - [stack/sandbox-websocket-server](databricks#7588) [[Files changed](https://github.com/databricks/cli/pull/7588/files)] [MERGED] - [**stack/sandbox-ssh-port-forward**](databricks#7589) [[Files changed](https://github.com/databricks/cli/pull/7589/files)] ← _this PR_ - [stack/sandbox-ssh-setup](databricks#7590) [[Files changed](https://github.com/databricks/cli/pull/7590/files)] --------- ## What did you change, and why? Keep `databricks sandbox ssh` while removing its dependence on the Gateway SSH listener and key registry. Use reusable local keys, command execution for expiring public-key entries, and authenticated WebSocket forwarding through `/sandbox-proxy/`. The internal proxy prepares and starts the hidden sandbox server, pins its host key, authenticates with its invocation token, and cleans up its command. Cache the Linux executable by architecture and version (content for development builds), retry uploads with a seekable body, and install uploads atomically. Retain at most eight unleased cached releases and expire unleased releases after 30 days. A hard link in each session directory protects its executable during startup and use. Serialize abandoned-session cleanup, and retain unexpected-failure logs while cleaning up successful or cancelled sessions. Renew WebSocket and command-output streams every 25 minutes. Negotiate the existing resume protocol through server readiness and reauthenticate each replacement WebSocket. Reattach transiently interrupted command output using the same command ID and sequence number, with a 60-second recovery budget that accommodates 30-second server heartbeats. Report startup progress on stderr and disable inherited OpenSSH connection multiplexing. Preserve writable inherited command-output pipes by duplicating their descriptors. Keep startup diagnostics visible and direct subsequent server logs to a session file. Keep the implementation under `experimental/ssh/sandbox` and reuse existing SSH helpers at their current paths. This requires workspace sandbox HTTP/WebSocket forwarding. ## How do you know it works? Validated build `aab1071a423c` against a real staging sandbox. After the review fixes, repeated the SSH and file-transfer checks below on `69946b9e5164` in a freshly created staging sandbox; all passed. Latest review revision `12e4dd1c38e6` passed the SSH and file-transfer operations below in another freshly created staging sandbox. It also passed: - Stop the sandbox through the API, then reconnect with `databricks sandbox ssh`: the CLI reports startup progress and the remote command succeeds. - Interrupt this invocation’s connections and block replacement connections for **eight seconds**: a **120-second** SSH session preserves its remote process and verifies **2,481,600 exact bytes**, then exits **0**. - Force the remote invocation server to exit unexpectedly: the CLI reports the retained session-directory path, and command execution confirms that its diagnostic log remains readable. The validation sandbox was deleted afterward. Local coverage includes 16 concurrent failed-start attempts, a complete SDK upload replay after HTTP 503, cache pruning with active and starting sessions, concurrent session cleanup under umask `002`, a long inherited OpenSSH `ControlPath`, 60-second output retry exhaustion, a silent first 30 seconds, and healthy output after 61 seconds. The new Python test harnesses also pass with system Python 3.6. **SSH and file transfer** Invoked the rebuilt CLI: ```bash "$CLI" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- true ``` Temporarily placed an `ssh` wrapper on `PATH` to record the arguments generated by the CLI, then execute system OpenSSH. Reused those same arguments—including the client identity, pinned host-key settings, and CLI `ProxyCommand`—for native SSH, SCP, and SFTP. The connection settings were equivalent to: ```bash SSH_OPTIONS=( -i "$KEY_FILE" -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o ConnectTimeout=900 -o StrictHostKeyChecking=yes -o "UserKnownHostsFile=$KNOWN_HOSTS_FILE" -o "HostKeyAlias=$SSH_IDENTITY" -o ServerAliveInterval=30 -o "ProxyCommand=$CLI sandbox ssh sandboxes/$SANDBOX_ID --proxy --profile $PROFILE --ssh-identity $SSH_IDENTITY" ) DESTINATION="sandbox-agent@$SSH_IDENTITY" ``` Paths and identity values above represent the actual CLI-generated values captured during the test. Ran the following operations, shown as shell equivalents of the Python subprocess calls: ```bash # Verify remote stdout and exit-status propagation. ssh "${SSH_OPTIONS[@]}" "$DESTINATION" \ 'printf NATIVE_SSH_OK; exit 23' # Verify remote terminal allocation. ssh "${SSH_OPTIONS[@]}" -tt "$DESTINATION" \ 'test -t 0 && printf PTY_OK' # Round-trip a binary payload through stdin/stdout. ssh "${SSH_OPTIONS[@]}" "$DESTINATION" cat \ < upload.bin > roundtrip.bin # Upload and download using native SCP. scp "${SSH_OPTIONS[@]}" upload.bin \ "$DESTINATION:/home/sandbox-agent/pr7589-live.bin" scp "${SSH_OPTIONS[@]}" \ "$DESTINATION:/home/sandbox-agent/pr7589-live.bin" downloaded.bin # Upload and download using native SFTP. sftp -b - "${SSH_OPTIONS[@]}" "$DESTINATION" <<EOF put upload.bin /home/sandbox-agent/pr7589-sftp.bin get /home/sandbox-agent/pr7589-sftp.bin sftp-downloaded.bin EOF ``` The payload was `bytes(range(256)) * 4096`—1 MiB containing every byte value. Python compared the round-trip stdout and both downloaded files byte for byte with the original. | Check | Recorded result | |---|---| | Remote command | `NATIVE_SSH_OK`; expected exit code **23** | | PTY | `PTY_OK`; exit **0** | | Binary stdin/stdout | Exact match | | SCP upload/download | Exact match | | SFTP upload/download | Exact match | | Remove test files | Exit **0** | Payload SHA-256: ```text fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83 ``` **Connection recovery and handover** An earlier build, `07981d228a81`, completed a 28-minute real-sandbox session. A local CONNECT proxy interrupted the actual connections and blocked replacement connections for two seconds. The verifier checked every output frame and confirmed unchanged remote process identity. ```json { "exit_code": 0, "elapsed_seconds": 1684.558, "verified_bytes": 34742400, "exact_bytes": true, "same_remote_pid": true, "fault_resume_verified": true, "planned_handover_verified": true } ``` Server logs confirmed two reattachments and the scheduled 25-minute handover. After the final startup-error fix, repeated forced-disconnect recovery on build `aab1071a423c`: **120 seconds, 2,481,600 verified bytes, unchanged remote process, successful reattachment, exit 0**. Repeated the SSH and file-transfer checks above on that build. The 25-minute handover was not rerun after that final correction. **Local checks** Formatting, checks, lint, sandbox/compute SSH tests, affected race tests, and Windows amd64/Darwin arm64/Linux arm64 cross-compilation passed. ### Startup recovery validation (2026-11-19) Tested this PR at `6ab5086fb2f078c21c056ee47d14b69f0abc0be2`. Its CLI SSH command passed against real sandboxes created through both the Gateway API and the new Sandboxes API. The matching PR3 build, `88a9af89871bdc738512594c2c95cfb16df6297c`, supplied Host configuration for native SSH/SCP/SFTP validation of the same transport. Both sandbox types passed stopped-sandbox startup, interactive native SSH, PTY, remote command/exit status, exact binary stdin/stdout, and native SCP/SFTP upload/download with a 1 MiB payload. Setup while stopped, repeated setup, overwrite protection, and cleanup also passed. Native connections used the generated configuration without hand-written identity or ProxyCommand options. ```sh "$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'printf CLI_E2E_OK' # Output: CLI_E2E_OK; exit 0. # After generating the native-tool configuration with the matching PR3 build: ssh -F "$SSH_CONFIG" my-sandbox scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin ``` Recovery tests used a local proxy connected to a real staging sandbox. Each native command was invoked once; recovery happened inside the CLI. | Injected failure | Previous build | Fixed transport | | --- | --- | --- | | ExecuteCommand unavailable before command registration | AttachCommand NotFound ended SSH | Retry with the same command ID; SSH and native SCP succeeded | | Response lost after real command registration | Not used as a negative control | Daemon rejected duplicate execution; attachment recovered the original command | | Initial forwarding HTTP 503 | Bad handshake ended SSH | Confirm termination, then use a fresh server; SSH and both SCP directions passed | | HTTP 503 after the remote WebSocket was accepted | Not used as a negative control | Retired the old invocation and connected through a fresh server | | More than one minute of setup before HTTP 503 | Not used as a negative control | Retried within the shared startup deadline | A separate 120-second real-session interruption test retained the remote process and verified all 2,481,600 bytes. The earlier upstream interruption itself could not be attributed from the available logs; these client recovery failures were reproduced and fixed. Regression tests cover safe same-ID execution recovery, refusal to restart a confirmed missing command, HTTP 429/503 recovery, immediate HTTP 403 failure, cancellation/cleanup, refusal to retry when server termination fails, and preserving stdin before connection establishment. Full sandbox/SSH suites, race tests, formatting/checks/linters, and local Isaac review pass. Native SCP used installed OpenSSH 8.2; SFTP was tested separately. ### Validation after nonblocking review fixes (2026-11-19) Tested a clean build of this PR at `c2d75cdea22149c44249e6ef1340058ab597da6c` against two newly created real staging sandboxes, one created through `POST /api/2.0/lakebox/sandboxes` and one through the public Sandboxes API, `POST /api/2.0/sandboxes`. Each PR/API combination used its own sandbox; binaries were checked against their commit versions and SHA-256 before and after testing. PR2 was tested independently, without `ssh-setup` or a PR3 binary. A temporary local `ssh` wrapper recorded the CLI-generated identity, host-key options, destination, and ProxyCommand, then executed system OpenSSH. Native SSH/SCP/SFTP reused those captured settings. The wrapper was used only to capture arguments; it did not replace the transport. | Check | Gateway-created | Public API-created | | --- | --- | --- | | CLI SSH and native SSH automatically start a stopped sandbox | Passed | Passed | | Remote stdout, login user, and expected exit 23 | Passed | Passed | | Interactive CLI and native SSH, typed commands, expected exit 29 | Passed | Passed | | Explicit PTY allocation through CLI and native SSH | Passed | Passed | | 1 MiB binary stdin/stdout through CLI and native SSH | Exact match | Exact match | | Native SCP upload/download and native SFTP upload/download | Exact match | Exact match | | Delete test sandbox and confirm it is missing; preserve personal SSH configuration | Passed | Passed | The PTY test launched each interactive command in a local pseudo-terminal, waited for the remote shell prompt, typed a unique marker, `id -un`, `pwd`, and `exit 29`, then checked the output and exit status. Binary tests used `bytes(range(256)) * 4096` and compared stdout and downloaded files byte for byte. Payload SHA-256: `fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83`. Timeout cleanup coverage confirms that startup deadlines and readiness failures retain session diagnostics, while success and explicit user cancellation remove the session directory. The full sandbox and compute SSH suites, affected race tests, formatting, checks, lint, and local Isaac review passed. ```sh # Direct CLI SSH; the test verified the expected remote exit status. "$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'id -un; exit 23' # For native tools, SSH_OPTIONS and DESTINATION below are captured from this # PR's own OpenSSH invocation, including its identity and ProxyCommand. ssh "${SSH_OPTIONS[@]}" "$DESTINATION" cat < upload.bin > roundtrip.bin scp "${SSH_OPTIONS[@]}" upload.bin "$DESTINATION:/tmp/upload.bin" scp "${SSH_OPTIONS[@]}" "$DESTINATION:/tmp/upload.bin" downloaded.bin sftp "${SSH_OPTIONS[@]}" -b - "$DESTINATION" <<EOF_SFTP put upload.bin /tmp/sftp.bin get /tmp/sftp.bin sftp-downloaded.bin EOF_SFTP ``` Commands above show the tested operations with workspace-specific values and unique remote file names replaced by variables. Installed OpenSSH 8.2 uses legacy SCP; SFTP was exercised separately. The earlier long-session handover results remain historical; this rerun covers the nonblocking fixes and both creation APIs. _This PR was written with Codex._ --------- Co-authored-by: Hsing Tang <HsingTang@users.noreply.github.com>
## 🥞 Stacked PR Use this [link](https://github.com/databricks/cli/pull/7590/files) to review incremental changes. - [stack/sandbox-websocket-server](databricks#7588) [[Files changed](https://github.com/databricks/cli/pull/7588/files)] [MERGED] - [stack/sandbox-ssh-port-forward](databricks#7589) [[Files changed](https://github.com/databricks/cli/pull/7589/files)] [MERGED] - [**stack/sandbox-ssh-setup**](databricks#7590) [[Files changed](https://github.com/databricks/cli/pull/7590/files)] ← _this PR_ --------- ## Changes Add `databricks sandbox ssh-setup SANDBOX_ID --name ALIAS --profile PROFILE` to write a local SSH Host configuration using the existing sandbox proxy mode. Native `ssh`, `scp`, and `sftp` can then connect using that alias. Setup requires an explicit Databricks profile, reuses local keys, retains a selected custom Databricks configuration file, and preserves unrelated SSH configuration. It does not start the sandbox. Sandbox SSH configuration helpers live alongside the other new sandbox implementation under `experimental/ssh/sandbox`. ## Why Support native SSH tools through the same authenticated forwarding path as `databricks sandbox ssh`. Interactive CLI SSH continues to work without a persistent Host entry or an explicit setup step. ## Tests Unit tests cover profile requirements, alias validation, idempotency, overwrite protection, and preservation of unrelated configuration. Installed OpenSSH parses the generated configuration. Acceptance tests exercise setup, repeated setup, saved connection settings, and invalid input. `./task test-sandbox test-exp-ssh`, affected race tests, and `./task fmt checks lint` pass. The package-layout change was reviewed locally with Isaac, with no blocking findings. ### Real-sandbox validation (2026-11-19) Passed against newly created staging sandboxes from both APIs: `POST /api/2.0/lakebox/sandboxes` and `POST /api/2.0/sandboxes`. Tested clean builds of this PR at `88a9af89871bdc738512594c2c95cfb16df6297c` and PR2 at `6ab5086fb2f078c21c056ee47d14b69f0abc0be2`. PR2's binary was tested directly for CLI SSH; this PR's binary generated the Host configuration used by native tools. Each sandbox was stopped before setup. Setup succeeded without starting it; the subsequent native SSH connection started it automatically. Tests used a separate SSH configuration file, preserving the user's normal configuration. Native commands used the generated alias, identity, host-key settings, and ProxyCommand without supplying transport arguments manually. ```sh # CLI SSH: run using the PR2 binary, which has no ssh-setup command. "$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'printf CLI_E2E_OK' # Output: CLI_E2E_OK; exit 0. # Native tools: generate configuration using the PR3 binary. "$CLI3" sandbox ssh-setup "$SANDBOX_ID" --name my-sandbox \ --profile "$PROFILE" --ssh-config "$SSH_CONFIG" ssh -G -F "$SSH_CONFIG" my-sandbox ssh -F "$SSH_CONFIG" my-sandbox 'printf PR7590_NATIVE_SSH_OK; id -un; exit 23' # Output includes PR7590_NATIVE_SSH_OK and sandbox-agent; expected exit 23. ssh -F "$SSH_CONFIG" my-sandbox # In the interactive shell: print a marker, run id -un and pwd, then exit 29. scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin sftp -F "$SSH_CONFIG" -b - my-sandbox <<EOF_SFTP put upload.bin /tmp/sftp-upload.bin get /tmp/sftp-upload.bin sftp-downloaded.bin EOF_SFTP ``` | Check | Gateway-created sandbox | API-created sandbox | | --- | --- | --- | | Setup while stopped; automatic start on connection | Passed | Passed | | OpenSSH configuration parsing and repeated setup | Passed; files unchanged on repeated setup | Passed; files unchanged on repeated setup | | CLI SSH, native command, interactive shell, PTY, exit status | Passed | Passed | | Binary stdin/stdout, native SCP upload/download, native SFTP upload/download | Passed; exact 1 MiB payload in each case | Passed; exact 1 MiB payload in each case | | Preserve unrelated configuration; reject alias overwrite; explicit overwrite; invalid alias/profile handling | Passed | Passed | | Delete test sandbox and remove test-owned aliases | Passed | Passed | Payload: `bytes(range(256)) * 4096`; SHA-256 `fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83`. The earlier live failures exposed recovery defects in PR2's transport, now fixed there. A local test proxy forwarded authenticated requests to a real sandbox while injecting startup failures. The previous build reproduced both visible failures; the tested revision recovered automatically, with no caller retries: - ExecuteCommand unavailable before registration: attach returned NotFound, then execution retried with the same command ID. - Lost response after real command registration: the daemon rejected duplicate execution and the client attached to the original command. - Initial forwarding HTTP 503: the CLI stopped the failed invocation and started a fresh server. Native SSH and both SCP directions completed with exact bytes. - HTTP 503 after the sandbox had already accepted the WebSocket: cleanup and a fresh invocation recovered without retrying a handover. - Setup lasting over one minute followed by HTTP 503: recovery still completed within the shared startup deadline. A separate 120-second real session survived a forced network interruption with the same remote process and all 2,481,600 stream bytes verified. The original upstream interruption could not be attributed from the available logs; the client failure modes were reproduced and fixed. Local sandbox and SSH suites, race checks, formatting, checks, and linters pass. Isaac's final local review has no remaining findings. The installed OpenSSH client was 8.2, so native SCP tested legacy SCP; native SFTP was tested separately. OpenSSH 9's default-SFTP SCP and desktop IDEs were not exercised. ### Validation after nonblocking review fixes (2026-11-19) Tested a clean build of this PR at `e7e1fc7696145f9a5b87f6b071bf2cd231e092bb` against two newly created real staging sandboxes, one created through `POST /api/2.0/lakebox/sandboxes` and one through the public Sandboxes API, `POST /api/2.0/sandboxes`. Each PR/API combination used its own sandbox; binaries were checked against their commit versions and SHA-256 before and after testing. PR3 was tested using its own binary for CLI SSH, `ssh-setup`, and every native-tool ProxyCommand. Setup ran while the sandbox was stopped and did not start it. Repeated setup left both configuration files unchanged; `ssh -G` parsed the saved profile, identity, host-key checks, and ProxyCommand. Tests also checked alias overwrite protection and explicit overwrite while preserving unrelated configuration. | Check | Gateway-created | Public API-created | | --- | --- | --- | | CLI SSH and native SSH automatically start a stopped sandbox | Passed | Passed | | Remote stdout, login user, and expected exit 23 | Passed | Passed | | Interactive CLI and native SSH, typed commands, expected exit 29 | Passed | Passed | | Explicit PTY allocation through CLI and native SSH | Passed | Passed | | 1 MiB binary stdin/stdout through CLI and native SSH | Exact match | Exact match | | Native SCP upload/download and native SFTP upload/download | Exact match | Exact match | | Delete test sandbox and confirm it is missing; preserve personal SSH configuration | Passed | Passed | The PTY test launched each interactive command in a local pseudo-terminal, waited for the remote shell prompt, typed a unique marker, `id -un`, `pwd`, and `exit 29`, then checked the output and exit status. Binary tests used `bytes(range(256)) * 4096` and compared stdout and downloaded files byte for byte. Payload SHA-256: `fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83`. ```sh "$CLI3" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'id -un; exit 23' "$CLI3" sandbox ssh-setup "$SANDBOX_ID" --name my-sandbox \ --profile "$PROFILE" --ssh-config "$SSH_CONFIG" ssh -G -F "$SSH_CONFIG" my-sandbox ssh -F "$SSH_CONFIG" my-sandbox ssh -F "$SSH_CONFIG" my-sandbox cat < upload.bin > roundtrip.bin scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin sftp -F "$SSH_CONFIG" -b - my-sandbox <<EOF_SFTP put upload.bin /tmp/sftp.bin get /tmp/sftp.bin sftp-downloaded.bin EOF_SFTP ``` The full sandbox and compute SSH suites, affected race tests, formatting, checks, lint, and local Isaac review passed. Commands above show the tested operations with workspace-specific values and unique remote file names replaced by variables. Installed OpenSSH 8.2 uses legacy SCP; SFTP was exercised separately. The earlier long-session handover results remain historical; this rerun covers the nonblocking fixes and both creation APIs. _This PR was written with Codex._ --------- Co-authored-by: Hsing Tang <HsingTang@users.noreply.github.com>
🥞 Stacked PR
Use this link to review incremental changes.
What did you change, and why?
Sandbox SSH needs a server that works through HTTP/WebSocket port forwarding without compute initialization. Add the hidden, internal
databricks sandbox servercommand while leaving the existing SSH command unchanged in this PR.Keep the implementation under
experimental/ssh/sandboxwith sandbox CODEOWNERS. Require an invocation token, validate forwarding ports, and allow one SSH session with the existing handover and versioned resume protocols. The shared proxy owns session admission and reports completion or process-start failure; invoking the process factory consumes the single session, while failed WebSocket upgrades can retry. The sandbox owns its first-connection deadline and shutdown. Preserve compute timeout defaults and remove unused proxy options.How do you know it works?
Sandbox and compute SSH suites, race tests, acceptance tests, and
./task fmt checks lintpass. Tests cover repeated WebSocket interruptions without losing bytes or restarting the SSH process, handover through a proxy with a connection lifetime limit, failed upgrades and process starts, startup deadlines, and cleanup. The SIGTERM acceptance test verifies a child that remains alive when stdin closes. A standalone native OpenSSH check built from this PR verifies that both SSH server processes exit when the server receives SIGTERM.The dependent client was also validated against a real sandbox, including SSH, SCP/SFTP, and a 28-minute session with forced-disconnect recovery and scheduled handover. Local Isaac review findings were addressed or checked against native OpenSSH behavior. The latest startup-failure correction passed repeated race tests with 16 concurrent dials, the full SSH suite, and a fresh local Isaac review. Its dependent client also passed fresh real-sandbox SSH/SCP/SFTP, stopped-sandbox startup, eight-second outage recovery, and failure-log retention checks.
Validation after nonblocking review fixes (2026-11-19)
At
cd5c2165022ec908f438fdf9085d2b5a3636f0a0, the PID stub publishes its completed file using a same-directory rename, and the concurrent startup test waits for all HTTP handlers before checking completion counts. The fixed test passed 3,000 repetitions with Go's race detector. The server acceptance script passed 1,000 runs with 16 concurrent workers. Formatting, lint, and local Isaac review passed with no remaining reported findings.This PR was written with Codex.