Repository navigation
Connect sandbox SSH through authenticated WebSocket port forwarding - #7589
Conversation
5cd926a to
417c077
Compare
✅ Integration test reportCommit: 49d6175
Top 6 slowest tests (at least 2 minutes):
|
d3e8539 to
e27bc56
Compare
5441e69 to
9434fda
Compare
e27bc56 to
36dc63c
Compare
…cks#7682) Apply `Dependencies` to Go module changes, not just Dependabot PRs. Examples: databricks#7653, databricks#7584, databricks#7589. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
anton-107
left a comment
There was a problem hiding this comment.
Requesting changes for the per-connection CLI upload: the two inline comments marked Blocking. The other two inline comments are questions about whether the route and the connection-lifetime behavior work the way you intend.
| return "", err | ||
| } | ||
| defer executable.Close() | ||
| destination := path.Join(setup.SessionDir, "databricks") |
There was a problem hiding this comment.
Blocking: this uploads the whole CLI on every connection, into a fresh session-* dir (setup.py:59). That's 53 MiB for a stripped release build and 75 MiB for a dev build.
- With
ssh-setup(Configure native SSH and SCP with sandbox ssh-setup #7590), everyssh,scp,rsyncand VS Code reconnect goes through this ProxyCommand, so each one pays the upload before it starts. On macOS and Windows clients,linuxExecutablealso downloads the release zip from GitHub again each time, because nothing is cached locally. databricks ssh connectuploads once per version and skips the upload when the file already exists (uploadReleasesininternal/client/releases.go).
Could this cache the binary per version, for example at ~/.databricks/sandbox-ssh/bin/<version>/databricks, and skip the upload when it's already there? Dev builds would still be re-uploaded, as they are for compute.
There was a problem hiding this comment.
The executable is now cached in persistent HOME by architecture and version; development/local-release builds use a content hash. A cache hit skips the download and upload. Installation verifies the upload checksum and atomically renames it under a lock, so concurrent connections cannot expose a partial executable. Tests cover reuse, changed development builds, concurrent installation, and recovery after a partial upload.
Updated in 637d42e.
| if not host_key.exists(): | ||
| subprocess.run(["ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(host_key)], check=True) | ||
| host_public = subprocess.check_output(["ssh-keygen", "-y", "-f", str(host_key)], universal_newlines=True).strip() | ||
| session = Path(tempfile.mkdtemp(prefix="session-", dir=base)) |
There was a problem hiding this comment.
Blocking: nothing ever removes old session dirs, and each one holds a copy of the CLI. The only cleanup is the client's deferred rm -rf (remote.go:132, 15 s timeout). That doesn't run if the ProxyCommand is killed (SIGKILL, an IDE killing its process tree), or if it can't reach the workspace when the session ends (network drop, laptop sleep). Idle shutdown on the server doesn't delete files either. So every unclean exit leaves a CLI copy in the user's persistent HOME for good.
Repro: start databricks sandbox ssh <id>, kill -9 the sandbox ssh ... --proxy process, then run du -sh ~/.databricks/sandbox-ssh/session-* in the sandbox.
With the binary cached per version (other comment), a session dir would only hold a few small files. Even so, they still pile up, so it's worth pruning old session-* dirs here, for example ones older than a day.
There was a problem hiding this comment.
Moved the executable out of session directories into the persistent release cache. Setup now prunes session-* directories older than 24 hours, skipping any with an active server.lock lease. The launcher retains that lock across exec and releases it on process exit, including SIGKILL. The regression test runs the actual launcher, verifies that an old active directory survives pruning, then verifies removal after exit.
Updated in 637d42e.
| return "", fmt.Errorf("invalid sandbox resource name %q", name) | ||
| } | ||
| endpoint.Scheme = "wss" | ||
| endpoint.Path = fmt.Sprintf("/sandbox-port-forward/%s/ports/%d/ssh", name, port) |
There was a problem hiding this comment.
Question: is /sandbox-port-forward/ the route you meant? Everything else I can find uses /sandbox-proxy/sandboxes/<id>/ports/<port>/:
sandbox forwardon main (Adddatabricks sandbox forward#7395,forwardRouteincmd/sandbox/forward.go), which was tested against staging- the Python SDK's
SandboxExt.forward - the request format in the port-forwarding design, where
/sandbox-port-forward/only shows up in the earlier proof-of-concept sections
If the gateway doesn't serve this prefix, every connection would fail at the WebSocket handshake, after the setup and the upload. TestForwardingURL pins this path, so the unit tests wouldn't catch it.
There was a problem hiding this comment.
Corrected the route to /sandbox-proxy/sandboxes/<id>/ports/<port>/ssh, matching sandbox forward's current prefix. Updated the URL assertions and authenticated WebSocket fixture. These local tests pass; they do not substitute for live workspace validation.
Updated in 637d42e.
There was a problem hiding this comment.
Confirmed the /sandbox-proxy/sandboxes/<id>/ports/<port>/ssh route against an actual staging sandbox using the rebuilt CLI. The WebSocket upgrades successfully and preserves the invocation-token header. The workspace host also serves the real command-execution gRPC calls used by this implementation. SSH, native SCP/SFTP, and local/SOCKS forwarding passed through that route.
| _, err := io.Copy(writer, input) | ||
| writer.CloseWithError(err) | ||
| }() | ||
| return proxy.RunClientProxy(ctx, reader, output, func() <-chan time.Time { return nil }, 30*time.Second, false, dial) |
There was a problem hiding this comment.
Question: is a session meant to end when the forwarding connection reaches its lifetime limit? As far as I can tell, the port-forwarding design caps a WebSocket on this route at about an hour (rclarey asked about this on #7588 too). This passes a nil handover tick and resumable=false, and the relay in #7588 answers a second connection with the same id with a 503 (server.go:66). So I'd expect every sandbox ssh and VS Code session to drop within the hour. The direct Gateway SSH path this replaces doesn't go through this route.
If that isn't intended: the compute tunnel already handles a limit like this with handover (requestHandoverTick on the client, handleExistingConnection on the server). Accepting a same-id handover in the relay should be enough, through an explicit option on proxy.NewProxyServer rather than checking query parameter names, plus a tick under the limit on the client. You can check it locally: put a proxy with a 60 s stream limit in front of sandbox server, and the ssh session dies at 60 s.
There was a problem hiding this comment.
Enabled same-session WebSocket handover and renew it every 25 minutes, with current workspace authentication on each dial. The server preserves the same sshd across handovers. The command-output stream also reattaches from its last sequence number on that interval, without restarting the server. Tests cover handover through an expiring reverse proxy, output reattachment/cancellation, and clean server-exit ordering. Unexpected-disconnect session resume remains disabled.
Updated in 637d42e.
There was a problem hiding this comment.
Validated a real 28-minute sandbox SSH session with the scheduled 25-minute handover. It preserved the same SSH/user process and all 34,742,400 verified stream bytes. Live testing also exposed an unexpected WebSocket interruption, so this revision now negotiates the existing resume-v2 protocol through readiness and reauthenticates each reattachment. The run recovered from the deliberate network interruption and a second interruption, with server logs confirming both reattachments and handover. Command-output recovery reattaches to the same command ID and last sequence number rather than restarting the server.
36dc63c to
cb5c0cf
Compare
637d42e to
c4637bc
Compare
cb5c0cf to
a6b20fd
Compare
anton-107
left a comment
There was a problem hiding this comment.
Requesting changes for the server logs that reach the user's terminal during a session, see the inline comment. Interactive sessions get corrupted on every 25-minute handover.
| return | ||
| } | ||
| if data := event.GetData(); data != nil { | ||
| if _, err := diagnostics.Write(data.GetStderr()); err != nil { |
There was a problem hiding this comment.
Blocking: the remote server's logs reach the user's terminal during the session.
readServerOutput copies the server's stderr into diagnostics on every event, including after readiness, and line 156 starts the server with --log-level info. diagnostics is the ProxyCommand's stderr, and OpenSSH passes that straight to the user's terminal, outside the SSH stream.
Interactive sessions get corrupted. While ssh runs, the local terminal is in raw mode. Every 25 minutes the handover logs Info: Client already connected, accepting handover and Info: Handover accepted, and those lines land wherever the cursor is, with a \n and no carriage return. In a shell that's a stray, staircased line. In vim, less, htop or tmux it overwrites part of the screen, and the remote app has no reason to redraw it. To the user it looks like a broken connection, and it repeats for as long as the session lasts.
It also shows up outside interactive use. Every ssh, scp, sftp and rsync, including through the ssh-setup aliases in #7590, starts with Info: start, Info: Sandbox SSH server ready on port … and Info: New connection accepted. A server stopped by TerminateCommand prints Error: context canceled.
Repro, no workspace needed: build this branch and run databricks sandbox server --sshd-config x --sshd <stub> --port 0 --log-level info, with a stub sshd that exits 0 for -t and runs exec sleep 300 otherwise. Open a WebSocket to /ssh?id=s1 with the readiness token, then a second one with the same id. The server's stderr, which is what this line forwards:
Info: start
Info: Sandbox SSH server ready on port 39651
Info: Starting proxy server for new connection, count: 1
Info: New connection accepted
Info: Client already connected, accepting handover
…
Info: Sandbox SSH server stopped
Error: context canceled
I didn't put a real ssh in front of it; the terminal effect follows from raw mode.
Suggested fix: keep forwarding stderr until readiness so startup failures still surface, then stop copying it. Write the server's log to a file in the session dir instead (for example --log-file <session>/server.log), so the lifecycle logging from the earlier round stays available for debugging.
There was a problem hiding this comment.
The server now runs with --log-file <session>/server.log, and readServerOutput stops forwarding remote stderr after readiness while retaining startup diagnostics. Added regression coverage for both startup diagnostics and suppression after readiness. Real sandbox remote-command, SCP, and SFTP runs complete without server lifecycle logs on stderr; the long-session handover logs remain in the server log.
|
Question: have you run |
c4637bc to
aab1071
Compare
a6b20fd to
219d06c
Compare
|
@anton-107 Yes—I created a real staging sandbox and ran the rebuilt CLI against the actual APIs in the live-validation request. That caught a real startup failure: command execution supplies a writable inherited stdout FIFO that cannot be reopened by the server user. The fix duplicates the inherited descriptor rather than reopening The live runs confirm the workspace gRPC Execute/Attach/Terminate calls and the WebSocket upgrade through A 28-minute session preserved its remote process and all 34,742,400 verified bytes through forced disconnection, another interruption, and the scheduled 25-minute handover. Separate transient gRPC output failures are handled by attaching to the same command ID at the last received sequence. The final startup-error correction was followed by another real forced-disconnect test and native SSH/file-transfer checks. Server lifecycle logs are kept in the session log rather than forwarded into the terminal after readiness. |
|
@review-oss-bot Added Go doc comments for |
| if _, err := os.Stat("/usr/sbin/sshd"); err != nil { | ||
| t.Skip("requires openssh-server") | ||
| } | ||
| home := t.TempDir() |
There was a problem hiding this comment.
This test fails on machines with umask 002: t.TempDir() comes out group-writable and setup.py rejects the HOME before any pruning runs. Chmod the temp home to 0700 so the test doesn't depend on the runner's umask.
There was a problem hiding this comment.
The pruning test now chmods its temporary HOME to 0700. The native sandbox tests also pass with umask 002.
|
|
||
| func (o *Output) reattach() (*ExecuteCommandResponse, error) { | ||
| var err error | ||
| for attempt := range 5 { |
There was a problem hiding this comment.
When the command-output stream drops, this loop gives up after five attach attempts, about 3.75s, and connectProxy then cancels the SSH proxy with "sandbox SSH server stopped" even though the server is still running and the WebSocket resume would keep retrying for 60s. Retry for at least the WebSocket resume budget (or don't end the session on a retryable output-stream error), and add a test with an outage longer than a few seconds.
There was a problem hiding this comment.
Command-output reattachment now uses a 60-second recovery budget, with capped backoff and immediate cancellation/permanent-error handling. The first response can use the full budget because server heartbeats are 30 seconds apart. Once output resumes, its stream is independent of the recovery deadline. Tests cover an output-only outage beyond the previous retry limit, full-budget exhaustion, a silent first 30 seconds, and continued output after 61 seconds. ExecuteCommand still runs only once and reattachment preserves its command ID and sequence.
| "-o", "ConnectTimeout=900", | ||
| "-o", "StrictHostKeyChecking=yes", "-o", "UserKnownHostsFile=" + quoteSSHOption(identity.KnownHostsPath), | ||
| "-o", "HostKeyAlias=" + identity.HostKeyAlias, "-o", "ServerAliveInterval=30", "-o", "ProxyCommand=" + proxyCommand, | ||
| "sandbox-agent@" + identity.HostKeyAlias, |
There was a problem hiding this comment.
The destination host is the 72-character sandbox-<sha256> alias, so for anyone whose ~/.ssh/config sets a ControlPath containing %h (for example ~/.ssh/cm-%r@%h:%p) ssh exits with "ControlPath too long" before it runs the ProxyCommand. Use a short destination host and keep the long value only in HostKeyAlias, or pass -o ControlPath=none.
There was a problem hiding this comment.
The generated OpenSSH arguments now include ControlPath=none. Each invocation owns its server, so it should not reuse a connection from the local multiplexing configuration. A real OpenSSH regression loads a long %r@%h:%p ControlPath and verifies that it reaches the configured ProxyCommand. The full alias remains in HostKeyAlias for host-key pinning.
| if err != nil { | ||
| return err | ||
| } | ||
| return execv.Execv(options) |
There was a problem hiding this comment.
sandbox ssh used to start a stopped sandbox itself and show a "Starting …" spinner; now the start (and the first-time CLI upload) happens inside the ProxyCommand, which prints nothing, so connecting to a stopped sandbox looks like a hang for as long as it takes to boot. Consider waiting for the sandbox with the spinner here before exec'ing ssh, or writing progress to the proxy's stderr.
There was a problem hiding this comment.
The proxy now reports progress on stderr before waiting for the sandbox to run and before checking or uploading its executable. Progress respects quiet mode and never enters the SSH byte stream. Local tests capture both messages; fresh live validation also stops a sandbox and reconnects through the CLI.
| if err != nil { | ||
| return err | ||
| } | ||
| defer cleanupRemoteDirectory(ctx, w, name, setup.SessionDir) |
There was a problem hiding this comment.
After readiness the server's stderr is dropped and its log goes to server.log in the session directory, which this cleanup deletes on every exit, including failures. A server that dies mid-session leaves only "server exited with status N"; consider keeping the directory, or printing the tail of the log, when the server exits non-zero.
There was a problem hiding this comment.
Unexpected failures now retain the session directory and server.log and print its location on stderr after failure. Successful sessions and cancellations still clean up; the existing abandoned-session pruning removes retained directories after 24 hours. The local fixture checks both retention and cleanup, and live validation deliberately stops the invocation server and verifies that its diagnostic log remains readable.
| } | ||
| } | ||
| temporary := path.Join(setup.SessionDir, "databricks.upload") | ||
| if err := w.Sandbox.UploadFile(ctx, sdk.UploadFileRequest{Parent: name, Path: temporary, Contents: io.NopCloser(bytes.NewReader(data))}); err != nil { |
There was a problem hiding this comment.
io.NopCloser hides the reader's Seek, and the SDK can only retry a request whose body it can rewind, so a single 429/504 or connection reset during this upload fails the connection instead of being retried. Pass a ReadCloser that still implements io.Seeker (for example a small type embedding *bytes.Reader with a no-op Close).
There was a problem hiding this comment.
Replaced io.NopCloser with a ReadCloser that preserves bytes.Reader.Seek. The SDK retry test consumes the first upload and returns 503, then verifies that the retried request contains the identical complete binary and that the installed file matches.
| } | ||
| cacheID = fmt.Sprintf("%x", sha256.Sum256(data)) | ||
| } | ||
| destination := path.Join(setup.Home, ".databricks", "sandbox-ssh", "releases", architecture+"-"+cacheID, "databricks") |
There was a problem hiding this comment.
Every CLI version, and every distinct development build, leaves its own copy of the binary (about 76 MiB for a dev build) under ~/.databricks/sandbox-ssh/releases in the sandbox's persistent home, and nothing ever removes old ones. Consider pruning old entries, as setup.py already does for stale session directories.
There was a problem hiding this comment.
Added cache pruning under a shared install lock: keep up to eight unleased releases and expire unleased entries after 30 days. Each session pins its executable with a hard link before the cache operation returns, protecting both an active server and the check-to-exec interval; session cleanup releases the pin. Tests cover age/count limits, multiple versions/architectures, current and active/starting releases, concurrent access, and release after session cleanup. Leased entries may temporarily exceed the cache limit.
| with open(old_session / "server.lock", "a") as lock: | ||
| fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) | ||
| shutil.rmtree(old_session) | ||
| except (BlockingIOError, FileNotFoundError): |
There was a problem hiding this comment.
If two connections run this pruning at the same time, one can recreate server.lock in a directory the other is removing, and rmtree then raises OSError: Directory not empty, which isn't caught, so that connection fails with a traceback. Catch OSError here, or do the pruning under a shared lock.
There was a problem hiding this comment.
Session pruning now holds a persistent shared sessions.lock, so concurrent setup processes cannot recreate server.lock inside a directory another pruner is deleting. Parallel setup coverage verifies removal of abandoned directories while retaining an active server and its executable pin.
| } | ||
| currentStatus = stopped.Status | ||
| s.Update("Starting " + id + "…") | ||
| if id == "" || strings.ContainsAny(id, "/\\?#") || strings.ContainsFunc(id, unicode.IsSpace) { |
There was a problem hiding this comment.
validateSandboxID in resolve.go already rejects IDs that aren't a single path segment, including . and .., which this inline check lets through into the /api/2.0/sandboxes/... and /sandbox-proxy/... paths. Call it here and keep only the extra ?, # and whitespace checks.
There was a problem hiding this comment.
ResolveSSHName now calls validateSandboxID, retaining the extra query, fragment, whitespace, and empty-ID checks. Added dot segments, prefixed dot segments, separators, and query/whitespace cases; the test verifies that invalid selections never reach a sandbox API request.
69946b9 to
12e4dd1
Compare
anton-107
left a comment
There was a problem hiding this comment.
My earlier blocking comments are addressed, and I checked each fix in the code:
- The CLI is now cached per architecture and version, and a cache hit skips the upload (
releases.go:54). - Session dirs older than 24 h are pruned unless their
server.locklease is held (setup.py:63). - The route is
/sandbox-proxy/…/ports/<port>/ssh, and the 25-minute handover plus resume v2 covers the WebSocket lifetime limit. - Server logs go to
<session>/server.log, and stderr stops being forwarded after readiness, so nothing reaches the terminal mid-session.
Builds, and go test -race passes for experimental/ssh/sandbox/..., cmd/sandbox/... and experimental/ssh/internal/.... The one failure there (internal/vscode) also fails on main.
Two non-blocking notes:
cleanupRemoteDirectory(remote.go:322) treats the 10-minute startup timeout like a cancellation and deletesserver.log, although that's the failure the log is most useful for.- This changes a released command:
sandbox sshloses--portand now needspython3andopenssh-serverin the sandbox. It might be worth saying so in the.nextchangesentry.
LGTM.
6ab5086 to
c2d75cd
Compare
|
@anton-107 Addressed both nonblocking notes in c2d75cd. Cleanup now uses context.Cause: startup/readiness timeouts retain the session directory and server.log, while success and explicit user cancellation still clean up. Tests cover these paths, and the cancellation fixture now calls cancel rather than using a deadline. The release note explicitly states removal of --port and the Python 3/OpenSSH server requirements in the sandbox environment. This PR's own binary passed fresh real-sandbox CLI/native SSH, PTY, exit status, 1 MiB binary input/output, SCP, and SFTP through both the Gateway and public API creation paths. No PR3 binary was used for PR2 validation. Commands and results are in the updated test plan. Local suites, race tests, formatting, lint, and Isaac review pass. |
c2d75cd to
49d6175
Compare
❌ Integration test reportCommit: 7f25e1e
27 interesting tests: 24 RECOVERED, 3 flaky
Top 50 slowest tests (at least 2 minutes):
|
## 🥞 Stacked PR Use this [link](https://github.com/databricks/cli/pull/7588/files) to review incremental changes. - [**stack/sandbox-websocket-server**](databricks#7588) [[Files changed](https://github.com/databricks/cli/pull/7588/files)] ← _this PR_ - [stack/sandbox-ssh-port-forward](databricks#7589) [[Files changed](https://github.com/databricks/cli/pull/7589/files)] - [stack/sandbox-ssh-setup](databricks#7590) [[Files changed](https://github.com/databricks/cli/pull/7590/files)] --------- ## What did you change, and why? Sandbox SSH needs a server that works through HTTP/WebSocket port forwarding without compute initialization. Add the hidden, internal `databricks sandbox server` command while leaving the existing SSH command unchanged in this PR. Keep the implementation under `experimental/ssh/sandbox` with sandbox CODEOWNERS. Require an invocation token, validate forwarding ports, and allow one SSH session with the existing handover and versioned resume protocols. The shared proxy owns session admission and reports completion or process-start failure; invoking the process factory consumes the single session, while failed WebSocket upgrades can retry. The sandbox owns its first-connection deadline and shutdown. Preserve compute timeout defaults and remove unused proxy options. ## How do you know it works? Sandbox and compute SSH suites, race tests, acceptance tests, and `./task fmt checks lint` pass. Tests cover repeated WebSocket interruptions without losing bytes or restarting the SSH process, handover through a proxy with a connection lifetime limit, failed upgrades and process starts, startup deadlines, and cleanup. The SIGTERM acceptance test verifies a child that remains alive when stdin closes. A standalone native OpenSSH check built from this PR verifies that both SSH server processes exit when the server receives SIGTERM. The dependent client was also validated against a real sandbox, including SSH, SCP/SFTP, and a 28-minute session with forced-disconnect recovery and scheduled handover. Local Isaac review findings were addressed or checked against native OpenSSH behavior. The latest startup-failure correction passed repeated race tests with 16 concurrent dials, the full SSH suite, and a fresh local Isaac review. Its dependent client also passed fresh real-sandbox SSH/SCP/SFTP, stopped-sandbox startup, eight-second outage recovery, and failure-log retention checks. ### Validation after nonblocking review fixes (2026-11-19) At `cd5c2165022ec908f438fdf9085d2b5a3636f0a0`, the PID stub publishes its completed file using a same-directory rename, and the concurrent startup test waits for all HTTP handlers before checking completion counts. The fixed test passed 3,000 repetitions with Go's race detector. The server acceptance script passed 1,000 runs with 16 concurrent workers. Formatting, lint, and local Isaac review passed with no remaining reported findings. _This PR was written with Codex._ --------- Co-authored-by: Hsing Tang <HsingTang@users.noreply.github.com>
## 🥞 Stacked PR Use this [link](https://github.com/databricks/cli/pull/7590/files) to review incremental changes. - [stack/sandbox-websocket-server](databricks#7588) [[Files changed](https://github.com/databricks/cli/pull/7588/files)] [MERGED] - [stack/sandbox-ssh-port-forward](databricks#7589) [[Files changed](https://github.com/databricks/cli/pull/7589/files)] [MERGED] - [**stack/sandbox-ssh-setup**](databricks#7590) [[Files changed](https://github.com/databricks/cli/pull/7590/files)] ← _this PR_ --------- ## Changes Add `databricks sandbox ssh-setup SANDBOX_ID --name ALIAS --profile PROFILE` to write a local SSH Host configuration using the existing sandbox proxy mode. Native `ssh`, `scp`, and `sftp` can then connect using that alias. Setup requires an explicit Databricks profile, reuses local keys, retains a selected custom Databricks configuration file, and preserves unrelated SSH configuration. It does not start the sandbox. Sandbox SSH configuration helpers live alongside the other new sandbox implementation under `experimental/ssh/sandbox`. ## Why Support native SSH tools through the same authenticated forwarding path as `databricks sandbox ssh`. Interactive CLI SSH continues to work without a persistent Host entry or an explicit setup step. ## Tests Unit tests cover profile requirements, alias validation, idempotency, overwrite protection, and preservation of unrelated configuration. Installed OpenSSH parses the generated configuration. Acceptance tests exercise setup, repeated setup, saved connection settings, and invalid input. `./task test-sandbox test-exp-ssh`, affected race tests, and `./task fmt checks lint` pass. The package-layout change was reviewed locally with Isaac, with no blocking findings. ### Real-sandbox validation (2026-11-19) Passed against newly created staging sandboxes from both APIs: `POST /api/2.0/lakebox/sandboxes` and `POST /api/2.0/sandboxes`. Tested clean builds of this PR at `88a9af89871bdc738512594c2c95cfb16df6297c` and PR2 at `6ab5086fb2f078c21c056ee47d14b69f0abc0be2`. PR2's binary was tested directly for CLI SSH; this PR's binary generated the Host configuration used by native tools. Each sandbox was stopped before setup. Setup succeeded without starting it; the subsequent native SSH connection started it automatically. Tests used a separate SSH configuration file, preserving the user's normal configuration. Native commands used the generated alias, identity, host-key settings, and ProxyCommand without supplying transport arguments manually. ```sh # CLI SSH: run using the PR2 binary, which has no ssh-setup command. "$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'printf CLI_E2E_OK' # Output: CLI_E2E_OK; exit 0. # Native tools: generate configuration using the PR3 binary. "$CLI3" sandbox ssh-setup "$SANDBOX_ID" --name my-sandbox \ --profile "$PROFILE" --ssh-config "$SSH_CONFIG" ssh -G -F "$SSH_CONFIG" my-sandbox ssh -F "$SSH_CONFIG" my-sandbox 'printf PR7590_NATIVE_SSH_OK; id -un; exit 23' # Output includes PR7590_NATIVE_SSH_OK and sandbox-agent; expected exit 23. ssh -F "$SSH_CONFIG" my-sandbox # In the interactive shell: print a marker, run id -un and pwd, then exit 29. scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin sftp -F "$SSH_CONFIG" -b - my-sandbox <<EOF_SFTP put upload.bin /tmp/sftp-upload.bin get /tmp/sftp-upload.bin sftp-downloaded.bin EOF_SFTP ``` | Check | Gateway-created sandbox | API-created sandbox | | --- | --- | --- | | Setup while stopped; automatic start on connection | Passed | Passed | | OpenSSH configuration parsing and repeated setup | Passed; files unchanged on repeated setup | Passed; files unchanged on repeated setup | | CLI SSH, native command, interactive shell, PTY, exit status | Passed | Passed | | Binary stdin/stdout, native SCP upload/download, native SFTP upload/download | Passed; exact 1 MiB payload in each case | Passed; exact 1 MiB payload in each case | | Preserve unrelated configuration; reject alias overwrite; explicit overwrite; invalid alias/profile handling | Passed | Passed | | Delete test sandbox and remove test-owned aliases | Passed | Passed | Payload: `bytes(range(256)) * 4096`; SHA-256 `fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83`. The earlier live failures exposed recovery defects in PR2's transport, now fixed there. A local test proxy forwarded authenticated requests to a real sandbox while injecting startup failures. The previous build reproduced both visible failures; the tested revision recovered automatically, with no caller retries: - ExecuteCommand unavailable before registration: attach returned NotFound, then execution retried with the same command ID. - Lost response after real command registration: the daemon rejected duplicate execution and the client attached to the original command. - Initial forwarding HTTP 503: the CLI stopped the failed invocation and started a fresh server. Native SSH and both SCP directions completed with exact bytes. - HTTP 503 after the sandbox had already accepted the WebSocket: cleanup and a fresh invocation recovered without retrying a handover. - Setup lasting over one minute followed by HTTP 503: recovery still completed within the shared startup deadline. A separate 120-second real session survived a forced network interruption with the same remote process and all 2,481,600 stream bytes verified. The original upstream interruption could not be attributed from the available logs; the client failure modes were reproduced and fixed. Local sandbox and SSH suites, race checks, formatting, checks, and linters pass. Isaac's final local review has no remaining findings. The installed OpenSSH client was 8.2, so native SCP tested legacy SCP; native SFTP was tested separately. OpenSSH 9's default-SFTP SCP and desktop IDEs were not exercised. ### Validation after nonblocking review fixes (2026-11-19) Tested a clean build of this PR at `e7e1fc7696145f9a5b87f6b071bf2cd231e092bb` against two newly created real staging sandboxes, one created through `POST /api/2.0/lakebox/sandboxes` and one through the public Sandboxes API, `POST /api/2.0/sandboxes`. Each PR/API combination used its own sandbox; binaries were checked against their commit versions and SHA-256 before and after testing. PR3 was tested using its own binary for CLI SSH, `ssh-setup`, and every native-tool ProxyCommand. Setup ran while the sandbox was stopped and did not start it. Repeated setup left both configuration files unchanged; `ssh -G` parsed the saved profile, identity, host-key checks, and ProxyCommand. Tests also checked alias overwrite protection and explicit overwrite while preserving unrelated configuration. | Check | Gateway-created | Public API-created | | --- | --- | --- | | CLI SSH and native SSH automatically start a stopped sandbox | Passed | Passed | | Remote stdout, login user, and expected exit 23 | Passed | Passed | | Interactive CLI and native SSH, typed commands, expected exit 29 | Passed | Passed | | Explicit PTY allocation through CLI and native SSH | Passed | Passed | | 1 MiB binary stdin/stdout through CLI and native SSH | Exact match | Exact match | | Native SCP upload/download and native SFTP upload/download | Exact match | Exact match | | Delete test sandbox and confirm it is missing; preserve personal SSH configuration | Passed | Passed | The PTY test launched each interactive command in a local pseudo-terminal, waited for the remote shell prompt, typed a unique marker, `id -un`, `pwd`, and `exit 29`, then checked the output and exit status. Binary tests used `bytes(range(256)) * 4096` and compared stdout and downloaded files byte for byte. Payload SHA-256: `fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83`. ```sh "$CLI3" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'id -un; exit 23' "$CLI3" sandbox ssh-setup "$SANDBOX_ID" --name my-sandbox \ --profile "$PROFILE" --ssh-config "$SSH_CONFIG" ssh -G -F "$SSH_CONFIG" my-sandbox ssh -F "$SSH_CONFIG" my-sandbox ssh -F "$SSH_CONFIG" my-sandbox cat < upload.bin > roundtrip.bin scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin sftp -F "$SSH_CONFIG" -b - my-sandbox <<EOF_SFTP put upload.bin /tmp/sftp.bin get /tmp/sftp.bin sftp-downloaded.bin EOF_SFTP ``` The full sandbox and compute SSH suites, affected race tests, formatting, checks, lint, and local Isaac review passed. Commands above show the tested operations with workspace-specific values and unique remote file names replaced by variables. Installed OpenSSH 8.2 uses legacy SCP; SFTP was exercised separately. The earlier long-session handover results remain historical; this rerun covers the nonblocking fixes and both creation APIs. _This PR was written with Codex._ --------- Co-authored-by: Hsing Tang <HsingTang@users.noreply.github.com>
🥞 Stacked PR
Use this link to review incremental changes.
What did you change, and why?
Keep
databricks sandbox sshwhile removing its dependence on the Gateway SSH listener and key registry. Use reusable local keys, command execution for expiring public-key entries, and authenticated WebSocket forwarding through/sandbox-proxy/.The internal proxy prepares and starts the hidden sandbox server, pins its host key, authenticates with its invocation token, and cleans up its command. Cache the Linux executable by architecture and version (content for development builds), retry uploads with a seekable body, and install uploads atomically. Retain at most eight unleased cached releases and expire unleased releases after 30 days. A hard link in each session directory protects its executable during startup and use. Serialize abandoned-session cleanup, and retain unexpected-failure logs while cleaning up successful or cancelled sessions.
Renew WebSocket and command-output streams every 25 minutes. Negotiate the existing resume protocol through server readiness and reauthenticate each replacement WebSocket. Reattach transiently interrupted command output using the same command ID and sequence number, with a 60-second recovery budget that accommodates 30-second server heartbeats. Report startup progress on stderr and disable inherited OpenSSH connection multiplexing. Preserve writable inherited command-output pipes by duplicating their descriptors. Keep startup diagnostics visible and direct subsequent server logs to a session file.
Keep the implementation under
experimental/ssh/sandboxand reuse existing SSH helpers at their current paths. This requires workspace sandbox HTTP/WebSocket forwarding.How do you know it works?
Validated build
aab1071a423cagainst a real staging sandbox. After the review fixes, repeated the SSH and file-transfer checks below on69946b9e5164in a freshly created staging sandbox; all passed.Latest review revision
12e4dd1c38e6passed the SSH and file-transfer operations below in another freshly created staging sandbox. It also passed:databricks sandbox ssh: the CLI reports startup progress and the remote command succeeds.The validation sandbox was deleted afterward. Local coverage includes 16 concurrent failed-start attempts, a complete SDK upload replay after HTTP 503, cache pruning with active and starting sessions, concurrent session cleanup under umask
002, a long inherited OpenSSHControlPath, 60-second output retry exhaustion, a silent first 30 seconds, and healthy output after 61 seconds. The new Python test harnesses also pass with system Python 3.6.SSH and file transfer
Invoked the rebuilt CLI:
Temporarily placed an
sshwrapper onPATHto record the arguments generated by the CLI, then execute system OpenSSH. Reused those same arguments—including the client identity, pinned host-key settings, and CLIProxyCommand—for native SSH, SCP, and SFTP.The connection settings were equivalent to:
Paths and identity values above represent the actual CLI-generated values captured during the test.
Ran the following operations, shown as shell equivalents of the Python subprocess calls:
The payload was
bytes(range(256)) * 4096—1 MiB containing every byte value. Python compared the round-trip stdout and both downloaded files byte for byte with the original.NATIVE_SSH_OK; expected exit code 23PTY_OK; exit 0Payload SHA-256:
Connection recovery and handover
An earlier build,
07981d228a81, completed a 28-minute real-sandbox session. A local CONNECT proxy interrupted the actual connections and blocked replacement connections for two seconds. The verifier checked every output frame and confirmed unchanged remote process identity.{ "exit_code": 0, "elapsed_seconds": 1684.558, "verified_bytes": 34742400, "exact_bytes": true, "same_remote_pid": true, "fault_resume_verified": true, "planned_handover_verified": true }Server logs confirmed two reattachments and the scheduled 25-minute handover.
After the final startup-error fix, repeated forced-disconnect recovery on build
aab1071a423c: 120 seconds, 2,481,600 verified bytes, unchanged remote process, successful reattachment, exit 0. Repeated the SSH and file-transfer checks above on that build. The 25-minute handover was not rerun after that final correction.Local checks
Formatting, checks, lint, sandbox/compute SSH tests, affected race tests, and Windows amd64/Darwin arm64/Linux arm64 cross-compilation passed.
Startup recovery validation (2026-11-19)
Tested this PR at
6ab5086fb2f078c21c056ee47d14b69f0abc0be2. Its CLI SSH command passed against real sandboxes created through both the Gateway API and the new Sandboxes API. The matching PR3 build,88a9af89871bdc738512594c2c95cfb16df6297c, supplied Host configuration for native SSH/SCP/SFTP validation of the same transport.Both sandbox types passed stopped-sandbox startup, interactive native SSH, PTY, remote command/exit status, exact binary stdin/stdout, and native SCP/SFTP upload/download with a 1 MiB payload. Setup while stopped, repeated setup, overwrite protection, and cleanup also passed. Native connections used the generated configuration without hand-written identity or ProxyCommand options.
Recovery tests used a local proxy connected to a real staging sandbox. Each native command was invoked once; recovery happened inside the CLI.
A separate 120-second real-session interruption test retained the remote process and verified all 2,481,600 bytes. The earlier upstream interruption itself could not be attributed from the available logs; these client recovery failures were reproduced and fixed.
Regression tests cover safe same-ID execution recovery, refusal to restart a confirmed missing command, HTTP 429/503 recovery, immediate HTTP 403 failure, cancellation/cleanup, refusal to retry when server termination fails, and preserving stdin before connection establishment. Full sandbox/SSH suites, race tests, formatting/checks/linters, and local Isaac review pass. Native SCP used installed OpenSSH 8.2; SFTP was tested separately.
Validation after nonblocking review fixes (2026-11-19)
Tested a clean build of this PR at
c2d75cdea22149c44249e6ef1340058ab597da6cagainst two newly created real staging sandboxes, one created throughPOST /api/2.0/lakebox/sandboxesand one through the public Sandboxes API,POST /api/2.0/sandboxes. Each PR/API combination used its own sandbox; binaries were checked against their commit versions and SHA-256 before and after testing.PR2 was tested independently, without
ssh-setupor a PR3 binary. A temporary localsshwrapper recorded the CLI-generated identity, host-key options, destination, and ProxyCommand, then executed system OpenSSH. Native SSH/SCP/SFTP reused those captured settings. The wrapper was used only to capture arguments; it did not replace the transport.The PTY test launched each interactive command in a local pseudo-terminal, waited for the remote shell prompt, typed a unique marker,
id -un,pwd, andexit 29, then checked the output and exit status. Binary tests usedbytes(range(256)) * 4096and compared stdout and downloaded files byte for byte. Payload SHA-256:fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83.Timeout cleanup coverage confirms that startup deadlines and readiness failures retain session diagnostics, while success and explicit user cancellation remove the session directory. The full sandbox and compute SSH suites, affected race tests, formatting, checks, lint, and local Isaac review passed.
Commands above show the tested operations with workspace-specific values and unique remote file names replaced by variables. Installed OpenSSH 8.2 uses legacy SCP; SFTP was exercised separately. The earlier long-session handover results remain historical; this rerun covers the nonblocking fixes and both creation APIs.
This PR was written with Codex.