Skip to content

Connect sandbox SSH through authenticated WebSocket port forwarding - #7589

Merged
HsingTang merged 10 commits into
mainfrom
HsingTang/stack/sandbox-ssh-port-forward
Nov 20, 2026
Merged

HsingTang merged 10 commits into
mainfrom
HsingTang/stack/sandbox-ssh-port-forward

Conversation

@HsingTang

@HsingTang HsingTang commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR

Use this link to review incremental changes.


What did you change, and why?

Keep databricks sandbox ssh while removing its dependence on the Gateway SSH listener and key registry. Use reusable local keys, command execution for expiring public-key entries, and authenticated WebSocket forwarding through /sandbox-proxy/.

The internal proxy prepares and starts the hidden sandbox server, pins its host key, authenticates with its invocation token, and cleans up its command. Cache the Linux executable by architecture and version (content for development builds), retry uploads with a seekable body, and install uploads atomically. Retain at most eight unleased cached releases and expire unleased releases after 30 days. A hard link in each session directory protects its executable during startup and use. Serialize abandoned-session cleanup, and retain unexpected-failure logs while cleaning up successful or cancelled sessions.

Renew WebSocket and command-output streams every 25 minutes. Negotiate the existing resume protocol through server readiness and reauthenticate each replacement WebSocket. Reattach transiently interrupted command output using the same command ID and sequence number, with a 60-second recovery budget that accommodates 30-second server heartbeats. Report startup progress on stderr and disable inherited OpenSSH connection multiplexing. Preserve writable inherited command-output pipes by duplicating their descriptors. Keep startup diagnostics visible and direct subsequent server logs to a session file.

Keep the implementation under experimental/ssh/sandbox and reuse existing SSH helpers at their current paths. This requires workspace sandbox HTTP/WebSocket forwarding.

How do you know it works?

Validated build aab1071a423c against a real staging sandbox. After the review fixes, repeated the SSH and file-transfer checks below on 69946b9e5164 in a freshly created staging sandbox; all passed.

Latest review revision 12e4dd1c38e6 passed the SSH and file-transfer operations below in another freshly created staging sandbox. It also passed:

  • Stop the sandbox through the API, then reconnect with databricks sandbox ssh: the CLI reports startup progress and the remote command succeeds.
  • Interrupt this invocation’s connections and block replacement connections for eight seconds: a 120-second SSH session preserves its remote process and verifies 2,481,600 exact bytes, then exits 0.
  • Force the remote invocation server to exit unexpectedly: the CLI reports the retained session-directory path, and command execution confirms that its diagnostic log remains readable.

The validation sandbox was deleted afterward. Local coverage includes 16 concurrent failed-start attempts, a complete SDK upload replay after HTTP 503, cache pruning with active and starting sessions, concurrent session cleanup under umask 002, a long inherited OpenSSH ControlPath, 60-second output retry exhaustion, a silent first 30 seconds, and healthy output after 61 seconds. The new Python test harnesses also pass with system Python 3.6.

SSH and file transfer

Invoked the rebuilt CLI:

"$CLI" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- true

Temporarily placed an ssh wrapper on PATH to record the arguments generated by the CLI, then execute system OpenSSH. Reused those same arguments—including the client identity, pinned host-key settings, and CLI ProxyCommand—for native SSH, SCP, and SFTP.

The connection settings were equivalent to:

SSH_OPTIONS=(
  -i "$KEY_FILE"
  -o IdentitiesOnly=yes
  -o PreferredAuthentications=publickey
  -o ConnectTimeout=900
  -o StrictHostKeyChecking=yes
  -o "UserKnownHostsFile=$KNOWN_HOSTS_FILE"
  -o "HostKeyAlias=$SSH_IDENTITY"
  -o ServerAliveInterval=30
  -o "ProxyCommand=$CLI sandbox ssh sandboxes/$SANDBOX_ID --proxy --profile $PROFILE --ssh-identity $SSH_IDENTITY"
)
DESTINATION="sandbox-agent@$SSH_IDENTITY"

Paths and identity values above represent the actual CLI-generated values captured during the test.

Ran the following operations, shown as shell equivalents of the Python subprocess calls:

# Verify remote stdout and exit-status propagation.
ssh "${SSH_OPTIONS[@]}" "$DESTINATION" \
  'printf NATIVE_SSH_OK; exit 23'

# Verify remote terminal allocation.
ssh "${SSH_OPTIONS[@]}" -tt "$DESTINATION" \
  'test -t 0 && printf PTY_OK'

# Round-trip a binary payload through stdin/stdout.
ssh "${SSH_OPTIONS[@]}" "$DESTINATION" cat \
  < upload.bin > roundtrip.bin

# Upload and download using native SCP.
scp "${SSH_OPTIONS[@]}" upload.bin \
  "$DESTINATION:/home/sandbox-agent/pr7589-live.bin"

scp "${SSH_OPTIONS[@]}" \
  "$DESTINATION:/home/sandbox-agent/pr7589-live.bin" downloaded.bin

# Upload and download using native SFTP.
sftp -b - "${SSH_OPTIONS[@]}" "$DESTINATION" <<EOF
put upload.bin /home/sandbox-agent/pr7589-sftp.bin
get /home/sandbox-agent/pr7589-sftp.bin sftp-downloaded.bin
EOF

The payload was bytes(range(256)) * 4096—1 MiB containing every byte value. Python compared the round-trip stdout and both downloaded files byte for byte with the original.

Check Recorded result
Remote command NATIVE_SSH_OK; expected exit code 23
PTY PTY_OK; exit 0
Binary stdin/stdout Exact match
SCP upload/download Exact match
SFTP upload/download Exact match
Remove test files Exit 0

Payload SHA-256:

fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83

Connection recovery and handover

An earlier build, 07981d228a81, completed a 28-minute real-sandbox session. A local CONNECT proxy interrupted the actual connections and blocked replacement connections for two seconds. The verifier checked every output frame and confirmed unchanged remote process identity.

{
  "exit_code": 0,
  "elapsed_seconds": 1684.558,
  "verified_bytes": 34742400,
  "exact_bytes": true,
  "same_remote_pid": true,
  "fault_resume_verified": true,
  "planned_handover_verified": true
}

Server logs confirmed two reattachments and the scheduled 25-minute handover.

After the final startup-error fix, repeated forced-disconnect recovery on build aab1071a423c: 120 seconds, 2,481,600 verified bytes, unchanged remote process, successful reattachment, exit 0. Repeated the SSH and file-transfer checks above on that build. The 25-minute handover was not rerun after that final correction.

Local checks

Formatting, checks, lint, sandbox/compute SSH tests, affected race tests, and Windows amd64/Darwin arm64/Linux arm64 cross-compilation passed.

Startup recovery validation (2026-11-19)

Tested this PR at 6ab5086fb2f078c21c056ee47d14b69f0abc0be2. Its CLI SSH command passed against real sandboxes created through both the Gateway API and the new Sandboxes API. The matching PR3 build, 88a9af89871bdc738512594c2c95cfb16df6297c, supplied Host configuration for native SSH/SCP/SFTP validation of the same transport.

Both sandbox types passed stopped-sandbox startup, interactive native SSH, PTY, remote command/exit status, exact binary stdin/stdout, and native SCP/SFTP upload/download with a 1 MiB payload. Setup while stopped, repeated setup, overwrite protection, and cleanup also passed. Native connections used the generated configuration without hand-written identity or ProxyCommand options.

"$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'printf CLI_E2E_OK'
# Output: CLI_E2E_OK; exit 0.

# After generating the native-tool configuration with the matching PR3 build:
ssh -F "$SSH_CONFIG" my-sandbox
scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin
scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin

Recovery tests used a local proxy connected to a real staging sandbox. Each native command was invoked once; recovery happened inside the CLI.

Injected failure Previous build Fixed transport
ExecuteCommand unavailable before command registration AttachCommand NotFound ended SSH Retry with the same command ID; SSH and native SCP succeeded
Response lost after real command registration Not used as a negative control Daemon rejected duplicate execution; attachment recovered the original command
Initial forwarding HTTP 503 Bad handshake ended SSH Confirm termination, then use a fresh server; SSH and both SCP directions passed
HTTP 503 after the remote WebSocket was accepted Not used as a negative control Retired the old invocation and connected through a fresh server
More than one minute of setup before HTTP 503 Not used as a negative control Retried within the shared startup deadline

A separate 120-second real-session interruption test retained the remote process and verified all 2,481,600 bytes. The earlier upstream interruption itself could not be attributed from the available logs; these client recovery failures were reproduced and fixed.

Regression tests cover safe same-ID execution recovery, refusal to restart a confirmed missing command, HTTP 429/503 recovery, immediate HTTP 403 failure, cancellation/cleanup, refusal to retry when server termination fails, and preserving stdin before connection establishment. Full sandbox/SSH suites, race tests, formatting/checks/linters, and local Isaac review pass. Native SCP used installed OpenSSH 8.2; SFTP was tested separately.

Validation after nonblocking review fixes (2026-11-19)

Tested a clean build of this PR at c2d75cdea22149c44249e6ef1340058ab597da6c against two newly created real staging sandboxes, one created through POST /api/2.0/lakebox/sandboxes and one through the public Sandboxes API, POST /api/2.0/sandboxes. Each PR/API combination used its own sandbox; binaries were checked against their commit versions and SHA-256 before and after testing.

PR2 was tested independently, without ssh-setup or a PR3 binary. A temporary local ssh wrapper recorded the CLI-generated identity, host-key options, destination, and ProxyCommand, then executed system OpenSSH. Native SSH/SCP/SFTP reused those captured settings. The wrapper was used only to capture arguments; it did not replace the transport.

Check Gateway-created Public API-created
CLI SSH and native SSH automatically start a stopped sandbox Passed Passed
Remote stdout, login user, and expected exit 23 Passed Passed
Interactive CLI and native SSH, typed commands, expected exit 29 Passed Passed
Explicit PTY allocation through CLI and native SSH Passed Passed
1 MiB binary stdin/stdout through CLI and native SSH Exact match Exact match
Native SCP upload/download and native SFTP upload/download Exact match Exact match
Delete test sandbox and confirm it is missing; preserve personal SSH configuration Passed Passed

The PTY test launched each interactive command in a local pseudo-terminal, waited for the remote shell prompt, typed a unique marker, id -un, pwd, and exit 29, then checked the output and exit status. Binary tests used bytes(range(256)) * 4096 and compared stdout and downloaded files byte for byte. Payload SHA-256: fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83.

Timeout cleanup coverage confirms that startup deadlines and readiness failures retain session diagnostics, while success and explicit user cancellation remove the session directory. The full sandbox and compute SSH suites, affected race tests, formatting, checks, lint, and local Isaac review passed.

# Direct CLI SSH; the test verified the expected remote exit status.
"$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'id -un; exit 23'
# For native tools, SSH_OPTIONS and DESTINATION below are captured from this
# PR's own OpenSSH invocation, including its identity and ProxyCommand.
ssh "${SSH_OPTIONS[@]}" "$DESTINATION" cat < upload.bin > roundtrip.bin
scp "${SSH_OPTIONS[@]}" upload.bin "$DESTINATION:/tmp/upload.bin"
scp "${SSH_OPTIONS[@]}" "$DESTINATION:/tmp/upload.bin" downloaded.bin
sftp "${SSH_OPTIONS[@]}" -b - "$DESTINATION" <<EOF_SFTP
put upload.bin /tmp/sftp.bin
get /tmp/sftp.bin sftp-downloaded.bin
EOF_SFTP

Commands above show the tested operations with workspace-specific values and unique remote file names replaced by variables. Installed OpenSSH 8.2 uses legacy SCP; SFTP was exercised separately. The earlier long-session handover results remain historical; this rerun covers the nonblocking fixes and both creation APIs.

This PR was written with Codex.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 49d6175

Run: 44647132441

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 4:29
💚​ aws-windows-latest-4core-16gb 1 274 17 3:37
💚​ azure linux-2core-8gb 1 271 19 3:57
💚​ azure-windows-latest-4core-16gb 1 273 17 3:19
💚​ gcp linux-2core-8gb 1 272 19 4:15
💚​ gcp-windows-latest-4core-16gb 1 274 17 3:48
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:36 aws-windows-latest-4core-16gb TestAccept
3:34 gcp-windows-latest-4core-16gb TestAccept
3:18 azure-windows-latest-4core-16gb TestAccept
2:08 gcp linux-2core-8gb TestAccept
2:06 aws linux-2core-8gb TestAccept
2:01 azure linux-2core-8gb TestAccept

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from d3e8539 to e27bc56 Compare November 13, 2026 21:16
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch 2 times, most recently from 5441e69 to 9434fda Compare November 13, 2026 22:12
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from e27bc56 to 36dc63c Compare November 13, 2026 22:12
@pietern pietern added the Dependencies Pull requests that update a dependency file label Nov 16, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 16, 2026
…cks#7682)

Apply `Dependencies` to Go module changes, not just Dependabot PRs.

Examples: databricks#7653, databricks#7584, databricks#7589.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>

@anton-107 anton-107 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the per-connection CLI upload: the two inline comments marked Blocking. The other two inline comments are questions about whether the route and the connection-lifetime behavior work the way you intend.

Comment thread experimental/ssh/sandbox/releases.go Outdated
return "", err
}
defer executable.Close()
destination := path.Join(setup.SessionDir, "databricks")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this uploads the whole CLI on every connection, into a fresh session-* dir (setup.py:59). That's 53 MiB for a stripped release build and 75 MiB for a dev build.

  • With ssh-setup (Configure native SSH and SCP with sandbox ssh-setup #7590), every ssh, scp, rsync and VS Code reconnect goes through this ProxyCommand, so each one pays the upload before it starts. On macOS and Windows clients, linuxExecutable also downloads the release zip from GitHub again each time, because nothing is cached locally.
  • databricks ssh connect uploads once per version and skips the upload when the file already exists (uploadReleases in internal/client/releases.go).

Could this cache the binary per version, for example at ~/.databricks/sandbox-ssh/bin/<version>/databricks, and skip the upload when it's already there? Dev builds would still be re-uploaded, as they are for compute.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The executable is now cached in persistent HOME by architecture and version; development/local-release builds use a content hash. A cache hit skips the download and upload. Installation verifies the upload checksum and atomically renames it under a lock, so concurrent connections cannot expose a partial executable. Tests cover reuse, changed development builds, concurrent installation, and recovery after a partial upload.

Updated in 637d42e.

if not host_key.exists():
subprocess.run(["ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(host_key)], check=True)
host_public = subprocess.check_output(["ssh-keygen", "-y", "-f", str(host_key)], universal_newlines=True).strip()
session = Path(tempfile.mkdtemp(prefix="session-", dir=base))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: nothing ever removes old session dirs, and each one holds a copy of the CLI. The only cleanup is the client's deferred rm -rf (remote.go:132, 15 s timeout). That doesn't run if the ProxyCommand is killed (SIGKILL, an IDE killing its process tree), or if it can't reach the workspace when the session ends (network drop, laptop sleep). Idle shutdown on the server doesn't delete files either. So every unclean exit leaves a CLI copy in the user's persistent HOME for good.

Repro: start databricks sandbox ssh <id>, kill -9 the sandbox ssh ... --proxy process, then run du -sh ~/.databricks/sandbox-ssh/session-* in the sandbox.

With the binary cached per version (other comment), a session dir would only hold a few small files. Even so, they still pile up, so it's worth pruning old session-* dirs here, for example ones older than a day.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the executable out of session directories into the persistent release cache. Setup now prunes session-* directories older than 24 hours, skipping any with an active server.lock lease. The launcher retains that lock across exec and releases it on process exit, including SIGKILL. The regression test runs the actual launcher, verifies that an old active directory survives pruning, then verifies removal after exit.

Updated in 637d42e.

Comment thread experimental/ssh/sandbox/remote.go Outdated
return "", fmt.Errorf("invalid sandbox resource name %q", name)
}
endpoint.Scheme = "wss"
endpoint.Path = fmt.Sprintf("/sandbox-port-forward/%s/ports/%d/ssh", name, port)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: is /sandbox-port-forward/ the route you meant? Everything else I can find uses /sandbox-proxy/sandboxes/<id>/ports/<port>/:

  • sandbox forward on main (Add databricks sandbox forward #7395, forwardRoute in cmd/sandbox/forward.go), which was tested against staging
  • the Python SDK's SandboxExt.forward
  • the request format in the port-forwarding design, where /sandbox-port-forward/ only shows up in the earlier proof-of-concept sections

If the gateway doesn't serve this prefix, every connection would fail at the WebSocket handshake, after the setup and the upload. TestForwardingURL pins this path, so the unit tests wouldn't catch it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected the route to /sandbox-proxy/sandboxes/<id>/ports/<port>/ssh, matching sandbox forward's current prefix. Updated the URL assertions and authenticated WebSocket fixture. These local tests pass; they do not substitute for live workspace validation.

Updated in 637d42e.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed the /sandbox-proxy/sandboxes/<id>/ports/<port>/ssh route against an actual staging sandbox using the rebuilt CLI. The WebSocket upgrades successfully and preserves the invocation-token header. The workspace host also serves the real command-execution gRPC calls used by this implementation. SSH, native SCP/SFTP, and local/SOCKS forwarding passed through that route.

Comment thread experimental/ssh/sandbox/remote.go Outdated
_, err := io.Copy(writer, input)
writer.CloseWithError(err)
}()
return proxy.RunClientProxy(ctx, reader, output, func() <-chan time.Time { return nil }, 30*time.Second, false, dial)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: is a session meant to end when the forwarding connection reaches its lifetime limit? As far as I can tell, the port-forwarding design caps a WebSocket on this route at about an hour (rclarey asked about this on #7588 too). This passes a nil handover tick and resumable=false, and the relay in #7588 answers a second connection with the same id with a 503 (server.go:66). So I'd expect every sandbox ssh and VS Code session to drop within the hour. The direct Gateway SSH path this replaces doesn't go through this route.

If that isn't intended: the compute tunnel already handles a limit like this with handover (requestHandoverTick on the client, handleExistingConnection on the server). Accepting a same-id handover in the relay should be enough, through an explicit option on proxy.NewProxyServer rather than checking query parameter names, plus a tick under the limit on the client. You can check it locally: put a proxy with a 60 s stream limit in front of sandbox server, and the ssh session dies at 60 s.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enabled same-session WebSocket handover and renew it every 25 minutes, with current workspace authentication on each dial. The server preserves the same sshd across handovers. The command-output stream also reattaches from its last sequence number on that interval, without restarting the server. Tests cover handover through an expiring reverse proxy, output reattachment/cancellation, and clean server-exit ordering. Unexpected-disconnect session resume remains disabled.

Updated in 637d42e.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated a real 28-minute sandbox SSH session with the scheduled 25-minute handover. It preserved the same SSH/user process and all 34,742,400 verified stream bytes. Live testing also exposed an unexpected WebSocket interruption, so this revision now negotiates the existing resume-v2 protocol through readiness and reauthenticates each reattachment. The run recovered from the deliberate network interruption and a second interruption, with server logs confirming both reattachments and handover. Command-output recovery reattaches to the same command ID and last sequence number rather than restarting the server.

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from 36dc63c to cb5c0cf Compare November 18, 2026 05:45
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch 2 times, most recently from 637d42e to c4637bc Compare November 18, 2026 06:01
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from cb5c0cf to a6b20fd Compare November 18, 2026 06:01

@anton-107 anton-107 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the server logs that reach the user's terminal during a session, see the inline comment. Interactive sessions get corrupted on every 25-minute handover.

Comment thread experimental/ssh/sandbox/remote.go Outdated
return
}
if data := event.GetData(); data != nil {
if _, err := diagnostics.Write(data.GetStderr()); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the remote server's logs reach the user's terminal during the session.

readServerOutput copies the server's stderr into diagnostics on every event, including after readiness, and line 156 starts the server with --log-level info. diagnostics is the ProxyCommand's stderr, and OpenSSH passes that straight to the user's terminal, outside the SSH stream.

Interactive sessions get corrupted. While ssh runs, the local terminal is in raw mode. Every 25 minutes the handover logs Info: Client already connected, accepting handover and Info: Handover accepted, and those lines land wherever the cursor is, with a \n and no carriage return. In a shell that's a stray, staircased line. In vim, less, htop or tmux it overwrites part of the screen, and the remote app has no reason to redraw it. To the user it looks like a broken connection, and it repeats for as long as the session lasts.

It also shows up outside interactive use. Every ssh, scp, sftp and rsync, including through the ssh-setup aliases in #7590, starts with Info: start, Info: Sandbox SSH server ready on port … and Info: New connection accepted. A server stopped by TerminateCommand prints Error: context canceled.

Repro, no workspace needed: build this branch and run databricks sandbox server --sshd-config x --sshd <stub> --port 0 --log-level info, with a stub sshd that exits 0 for -t and runs exec sleep 300 otherwise. Open a WebSocket to /ssh?id=s1 with the readiness token, then a second one with the same id. The server's stderr, which is what this line forwards:

Info: start
Info: Sandbox SSH server ready on port 39651
Info: Starting proxy server for new connection, count: 1
Info: New connection accepted
Info: Client already connected, accepting handover
…
Info: Sandbox SSH server stopped
Error: context canceled

I didn't put a real ssh in front of it; the terminal effect follows from raw mode.

Suggested fix: keep forwarding stderr until readiness so startup failures still surface, then stop copying it. Write the server's log to a file in the session dir instead (for example --log-file <session>/server.log), so the lifecycle logging from the earlier round stays available for debugging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The server now runs with --log-file <session>/server.log, and readServerOutput stops forwarding remote stderr after readiness while retaining startup diagnostics. Added regression coverage for both startup diagnostics and suppression after readiness. Real sandbox remote-command, SCP, and SFTP runs complete without server lifecycle logs on stderr; the long-session handover logs remain in the server log.

@anton-107

Copy link
Copy Markdown
Contributor

Question: have you run databricks sandbox ssh from this branch against the real API? The local tests rely on three things only a live run shows: the command-execution service gives the server a pipe for stdout (openServerOutput refuses anything else), /sandbox-proxy/ passes X-Sandbox-SSH-Token through, and the workspace host serves the SandboxService gRPC calls. If any of them fails, every sandbox ssh fails, and this PR removes the Gateway path. If you haven't run it, are you OK releasing it as is?

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from c4637bc to aab1071 Compare November 19, 2026 05:15
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from a6b20fd to 219d06c Compare November 19, 2026 05:15
@HsingTang

Copy link
Copy Markdown
Contributor Author

@anton-107 Yes—I created a real staging sandbox and ran the rebuilt CLI against the actual APIs in the live-validation request.

That caught a real startup failure: command execution supplies a writable inherited stdout FIFO that cannot be reopened by the server user. The fix duplicates the inherited descriptor rather than reopening /proc/self/fd/1; regression tests cover the permissions and interruptible output writes.

The live runs confirm the workspace gRPC Execute/Attach/Terminate calls and the WebSocket upgrade through /sandbox-proxy/, including preservation of the invocation token. SSH commands and exit status, PTY, exact binary input/output, native SCP/SFTP upload/download, local/SOCKS forwarding, stopped-sandbox startup, persistent host identity, and relay/SSH-process cleanup passed.

A 28-minute session preserved its remote process and all 34,742,400 verified bytes through forced disconnection, another interruption, and the scheduled 25-minute handover. Separate transient gRPC output failures are handled by attaching to the same command ID at the last received sequence. The final startup-error correction was followed by another real forced-disconnect test and native SSH/file-transfer checks. Server lifecycle logs are kept in the session log rather than forwarded into the terminal after readiness.

@HsingTang
HsingTang marked this pull request as ready for review November 19, 2026 05:23
@HsingTang
HsingTang requested review from a team as code owners November 19, 2026 05:23

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New exported functions Name, OpenLocal, and OpenPublished in experimental/ssh/internal/releases/releases.go are missing doc comments (Go convention: "document functions with a doc comment that starts with the function name and ends with a period").

Comment thread experimental/ssh/sandbox/command_ssh.go Resolved
Comment thread experimental/ssh/sandbox/remote.go Resolved
@HsingTang
HsingTang requested a review from anton-107 November 19, 2026 05:55
@HsingTang

Copy link
Copy Markdown
Contributor Author

@review-oss-bot Added Go doc comments for releases.Name, releases.OpenLocal, and releases.OpenPublished, including caller ownership of the returned readers. Also addressed the two inline comments on inherited routing IDs and the WaitForRunning contract. Tests, lint, race checks, cross-builds, and fresh real-sandbox SSH/file-transfer validation pass.

if _, err := os.Stat("/usr/sbin/sshd"); err != nil {
t.Skip("requires openssh-server")
}
home := t.TempDir()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test fails on machines with umask 002: t.TempDir() comes out group-writable and setup.py rejects the HOME before any pruning runs. Chmod the temp home to 0700 so the test doesn't depend on the runner's umask.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pruning test now chmods its temporary HOME to 0700. The native sandbox tests also pass with umask 002.


func (o *Output) reattach() (*ExecuteCommandResponse, error) {
var err error
for attempt := range 5 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the command-output stream drops, this loop gives up after five attach attempts, about 3.75s, and connectProxy then cancels the SSH proxy with "sandbox SSH server stopped" even though the server is still running and the WebSocket resume would keep retrying for 60s. Retry for at least the WebSocket resume budget (or don't end the session on a retryable output-stream error), and add a test with an outage longer than a few seconds.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Command-output reattachment now uses a 60-second recovery budget, with capped backoff and immediate cancellation/permanent-error handling. The first response can use the full budget because server heartbeats are 30 seconds apart. Once output resumes, its stream is independent of the recovery deadline. Tests cover an output-only outage beyond the previous retry limit, full-budget exhaustion, a silent first 30 seconds, and continued output after 61 seconds. ExecuteCommand still runs only once and reattachment preserves its command ID and sequence.

"-o", "ConnectTimeout=900",
"-o", "StrictHostKeyChecking=yes", "-o", "UserKnownHostsFile=" + quoteSSHOption(identity.KnownHostsPath),
"-o", "HostKeyAlias=" + identity.HostKeyAlias, "-o", "ServerAliveInterval=30", "-o", "ProxyCommand=" + proxyCommand,
"sandbox-agent@" + identity.HostKeyAlias,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The destination host is the 72-character sandbox-<sha256> alias, so for anyone whose ~/.ssh/config sets a ControlPath containing %h (for example ~/.ssh/cm-%r@%h:%p) ssh exits with "ControlPath too long" before it runs the ProxyCommand. Use a short destination host and keep the long value only in HostKeyAlias, or pass -o ControlPath=none.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generated OpenSSH arguments now include ControlPath=none. Each invocation owns its server, so it should not reuse a connection from the local multiplexing configuration. A real OpenSSH regression loads a long %r@%h:%p ControlPath and verifies that it reaches the configured ProxyCommand. The full alias remains in HostKeyAlias for host-key pinning.

if err != nil {
return err
}
return execv.Execv(options)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sandbox ssh used to start a stopped sandbox itself and show a "Starting …" spinner; now the start (and the first-time CLI upload) happens inside the ProxyCommand, which prints nothing, so connecting to a stopped sandbox looks like a hang for as long as it takes to boot. Consider waiting for the sandbox with the spinner here before exec'ing ssh, or writing progress to the proxy's stderr.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proxy now reports progress on stderr before waiting for the sandbox to run and before checking or uploading its executable. Progress respects quiet mode and never enters the SSH byte stream. Local tests capture both messages; fresh live validation also stops a sandbox and reconnects through the CLI.

Comment thread experimental/ssh/sandbox/remote.go Outdated
if err != nil {
return err
}
defer cleanupRemoteDirectory(ctx, w, name, setup.SessionDir)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After readiness the server's stderr is dropped and its log goes to server.log in the session directory, which this cleanup deletes on every exit, including failures. A server that dies mid-session leaves only "server exited with status N"; consider keeping the directory, or printing the tail of the log, when the server exits non-zero.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unexpected failures now retain the session directory and server.log and print its location on stderr after failure. Successful sessions and cancellations still clean up; the existing abandoned-session pruning removes retained directories after 24 hours. The local fixture checks both retention and cleanup, and live validation deliberately stops the invocation server and verifies that its diagnostic log remains readable.

Comment thread experimental/ssh/sandbox/releases.go Outdated
}
}
temporary := path.Join(setup.SessionDir, "databricks.upload")
if err := w.Sandbox.UploadFile(ctx, sdk.UploadFileRequest{Parent: name, Path: temporary, Contents: io.NopCloser(bytes.NewReader(data))}); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

io.NopCloser hides the reader's Seek, and the SDK can only retry a request whose body it can rewind, so a single 429/504 or connection reset during this upload fails the connection instead of being retried. Pass a ReadCloser that still implements io.Seeker (for example a small type embedding *bytes.Reader with a no-op Close).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced io.NopCloser with a ReadCloser that preserves bytes.Reader.Seek. The SDK retry test consumes the first upload and returns 503, then verifies that the retried request contains the identical complete binary and that the installed file matches.

Comment thread experimental/ssh/sandbox/releases.go Outdated
}
cacheID = fmt.Sprintf("%x", sha256.Sum256(data))
}
destination := path.Join(setup.Home, ".databricks", "sandbox-ssh", "releases", architecture+"-"+cacheID, "databricks")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every CLI version, and every distinct development build, leaves its own copy of the binary (about 76 MiB for a dev build) under ~/.databricks/sandbox-ssh/releases in the sandbox's persistent home, and nothing ever removes old ones. Consider pruning old entries, as setup.py already does for stale session directories.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added cache pruning under a shared install lock: keep up to eight unleased releases and expire unleased entries after 30 days. Each session pins its executable with a hard link before the cache operation returns, protecting both an active server and the check-to-exec interval; session cleanup releases the pin. Tests cover age/count limits, multiple versions/architectures, current and active/starting releases, concurrent access, and release after session cleanup. Leased entries may temporarily exceed the cache limit.

Comment thread experimental/ssh/sandbox/setup.py Outdated
with open(old_session / "server.lock", "a") as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
shutil.rmtree(old_session)
except (BlockingIOError, FileNotFoundError):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If two connections run this pruning at the same time, one can recreate server.lock in a directory the other is removing, and rmtree then raises OSError: Directory not empty, which isn't caught, so that connection fails with a traceback. Catch OSError here, or do the pruning under a shared lock.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Session pruning now holds a persistent shared sessions.lock, so concurrent setup processes cannot recreate server.lock inside a directory another pruner is deleting. Parallel setup coverage verifies removal of abandoned directories while retaining an active server and its executable pin.

Comment thread cmd/sandbox/ssh.go Outdated
}
currentStatus = stopped.Status
s.Update("Starting " + id + "…")
if id == "" || strings.ContainsAny(id, "/\\?#") || strings.ContainsFunc(id, unicode.IsSpace) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validateSandboxID in resolve.go already rejects IDs that aren't a single path segment, including . and .., which this inline check lets through into the /api/2.0/sandboxes/... and /sandbox-proxy/... paths. Call it here and keep only the extra ?, # and whitespace checks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ResolveSSHName now calls validateSandboxID, retaining the extra query, fragment, whitespace, and empty-ID checks. Added dot segments, prefixed dot segments, separators, and query/whitespace cases; the test verifies that invalid selections never reach a sandbox API request.

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from 69946b9 to 12e4dd1 Compare November 19, 2026 16:18
@HsingTang
HsingTang requested a review from rclarey November 19, 2026 16:39

@anton-107 anton-107 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My earlier blocking comments are addressed, and I checked each fix in the code:

  • The CLI is now cached per architecture and version, and a cache hit skips the upload (releases.go:54).
  • Session dirs older than 24 h are pruned unless their server.lock lease is held (setup.py:63).
  • The route is /sandbox-proxy/…/ports/<port>/ssh, and the 25-minute handover plus resume v2 covers the WebSocket lifetime limit.
  • Server logs go to <session>/server.log, and stderr stops being forwarded after readiness, so nothing reaches the terminal mid-session.

Builds, and go test -race passes for experimental/ssh/sandbox/..., cmd/sandbox/... and experimental/ssh/internal/.... The one failure there (internal/vscode) also fails on main.

Two non-blocking notes:

  • cleanupRemoteDirectory (remote.go:322) treats the 10-minute startup timeout like a cancellation and deletes server.log, although that's the failure the log is most useful for.
  • This changes a released command: sandbox ssh loses --port and now needs python3 and openssh-server in the sandbox. It might be worth saying so in the .nextchanges entry.

LGTM.

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from 6ab5086 to c2d75cd Compare November 19, 2026 22:45
@HsingTang

Copy link
Copy Markdown
Contributor Author

@anton-107 Addressed both nonblocking notes in c2d75cd. Cleanup now uses context.Cause: startup/readiness timeouts retain the session directory and server.log, while success and explicit user cancellation still clean up. Tests cover these paths, and the cancellation fixture now calls cancel rather than using a deadline. The release note explicitly states removal of --port and the Python 3/OpenSSH server requirements in the sandbox environment.

This PR's own binary passed fresh real-sandbox CLI/native SSH, PTY, exit status, 1 MiB binary input/output, SCP, and SFTP through both the Gateway and public API creation paths. No PR3 binary was used for PR2 validation. Commands and results are in the updated test plan. Local suites, race tests, formatting, lint, and Isaac review pass.

Base automatically changed from HsingTang/stack/sandbox-websocket-server to main November 19, 2026 23:29
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from c2d75cd to 49d6175 Compare November 20, 2026 00:33
@HsingTang
HsingTang enabled auto-merge November 20, 2026 00:38
@HsingTang
HsingTang added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 7f25e1e Nov 20, 2026
38 checks passed
@HsingTang
HsingTang deleted the HsingTang/stack/sandbox-ssh-port-forward branch November 20, 2026 01:10
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: 7f25e1e

Run: 44651658901

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
🔄​ aws linux-2core-8gb 3 24 1619 1419 177:34
💚​ azure linux-2core-8gb 27 1599 1427 170:59
💚​ gcp linux-2core-8gb 27 1606 1425 183:18
27 interesting tests: 24 RECOVERED, 3 flaky
Test Name aws linux-2core-8gb azure linux-2core-8gb gcp linux-2core-8gb
🔄​ TestAccept 🔄​f 💚​R 💚​R
💚​ TestAccept/bundle/deployment/bind/alert 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/deployment/bind/alert/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/deployment/bind/alert/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/generate/alert 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/generate/alert/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/generate/alert/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/basic 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/basic/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/basic/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic/DMS= 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/resources/postgres_synced_tables/recreate 🔄​f 💚​R 💚​R
🔄​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS= 🔄​f 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS=true 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
10:06 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:55 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:52 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:54 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:49 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:09 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:37 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:50 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:49 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:45 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:42 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:28 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:22 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:11 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:59 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
5:52 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:42 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:39 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:37 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:33 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:07 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
4:45 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
4:39 azure linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:37 gcp linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:27 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
4:19 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
4:11 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
4:07 aws linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:03 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
3:47 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
3:47 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
3:45 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
3:41 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
3:40 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
3:31 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
3:30 gcp linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
3:27 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:26 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
3:26 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
3:25 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
3:18 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
3:17 aws linux-2core-8gb TestAccept/bundle/resources/apps/inline_config/DMS=true
3:14 gcp linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:12 azure linux-2core-8gb TestAccept/bundle/resources/apps/inline_config/DMS=true
3:11 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
3:05 azure linux-2core-8gb TestAccept/bundle/resources/apps/inline_config/DMS=
3:04 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=true
3:01 gcp linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
2:58 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
2:58 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=

idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
## 🥞 Stacked PR
Use this [link](https://github.com/databricks/cli/pull/7588/files) to
review incremental changes.
-
[**stack/sandbox-websocket-server**](databricks#7588)
[[Files changed](https://github.com/databricks/cli/pull/7588/files)] ←
_this PR_
-
[stack/sandbox-ssh-port-forward](databricks#7589)
[[Files changed](https://github.com/databricks/cli/pull/7589/files)]
- [stack/sandbox-ssh-setup](databricks#7590)
[[Files changed](https://github.com/databricks/cli/pull/7590/files)]

---------
## What did you change, and why?

Sandbox SSH needs a server that works through HTTP/WebSocket port
forwarding without compute initialization. Add the hidden, internal
`databricks sandbox server` command while leaving the existing SSH
command unchanged in this PR.

Keep the implementation under `experimental/ssh/sandbox` with sandbox
CODEOWNERS. Require an invocation token, validate forwarding ports, and
allow one SSH session with the existing handover and versioned resume
protocols. The shared proxy owns session admission and reports
completion or process-start failure; invoking the process factory
consumes the single session, while failed WebSocket upgrades can retry.
The sandbox owns its first-connection deadline and shutdown. Preserve
compute timeout defaults and remove unused proxy options.

## How do you know it works?

Sandbox and compute SSH suites, race tests, acceptance tests, and
`./task fmt checks lint` pass. Tests cover repeated WebSocket
interruptions without losing bytes or restarting the SSH process,
handover through a proxy with a connection lifetime limit, failed
upgrades and process starts, startup deadlines, and cleanup. The SIGTERM
acceptance test verifies a child that remains alive when stdin closes. A
standalone native OpenSSH check built from this PR verifies that both
SSH server processes exit when the server receives SIGTERM.

The dependent client was also validated against a real sandbox,
including SSH, SCP/SFTP, and a 28-minute session with forced-disconnect
recovery and scheduled handover. Local Isaac review findings were
addressed or checked against native OpenSSH behavior. The latest
startup-failure correction passed repeated race tests with 16 concurrent
dials, the full SSH suite, and a fresh local Isaac review. Its dependent
client also passed fresh real-sandbox SSH/SCP/SFTP, stopped-sandbox
startup, eight-second outage recovery, and failure-log retention checks.

### Validation after nonblocking review fixes (2026-11-19)

At `cd5c2165022ec908f438fdf9085d2b5a3636f0a0`, the PID stub publishes
its completed file using a same-directory rename, and the concurrent
startup test waits for all HTTP handlers before checking completion
counts. The fixed test passed 3,000 repetitions with Go's race detector.
The server acceptance script passed 1,000 runs with 16 concurrent
workers. Formatting, lint, and local Isaac review passed with no
remaining reported findings.

_This PR was written with Codex._

---------

Co-authored-by: Hsing Tang <HsingTang@users.noreply.github.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
## 🥞 Stacked PR
Use this [link](https://github.com/databricks/cli/pull/7590/files) to
review incremental changes.
-
[stack/sandbox-websocket-server](databricks#7588)
[[Files changed](https://github.com/databricks/cli/pull/7588/files)]
[MERGED]
-
[stack/sandbox-ssh-port-forward](databricks#7589)
[[Files changed](https://github.com/databricks/cli/pull/7589/files)]
[MERGED]
-
[**stack/sandbox-ssh-setup**](databricks#7590)
[[Files changed](https://github.com/databricks/cli/pull/7590/files)] ←
_this PR_

---------
## Changes

Add `databricks sandbox ssh-setup SANDBOX_ID --name ALIAS --profile
PROFILE` to write a local SSH Host configuration using the existing
sandbox proxy mode. Native `ssh`, `scp`, and `sftp` can then connect
using that alias. Setup requires an explicit Databricks profile, reuses
local keys, retains a selected custom Databricks configuration file, and
preserves unrelated SSH configuration. It does not start the sandbox.

Sandbox SSH configuration helpers live alongside the other new sandbox
implementation under `experimental/ssh/sandbox`.

## Why

Support native SSH tools through the same authenticated forwarding path
as `databricks sandbox ssh`. Interactive CLI SSH continues to work
without a persistent Host entry or an explicit setup step.

## Tests

Unit tests cover profile requirements, alias validation, idempotency,
overwrite protection, and preservation of unrelated configuration.
Installed OpenSSH parses the generated configuration. Acceptance tests
exercise setup, repeated setup, saved connection settings, and invalid
input.

`./task test-sandbox test-exp-ssh`, affected race tests, and `./task fmt
checks lint` pass. The package-layout change was reviewed locally with
Isaac, with no blocking findings.

### Real-sandbox validation (2026-11-19)

Passed against newly created staging sandboxes from both APIs: `POST
/api/2.0/lakebox/sandboxes` and `POST /api/2.0/sandboxes`. Tested clean
builds of this PR at `88a9af89871bdc738512594c2c95cfb16df6297c` and PR2
at `6ab5086fb2f078c21c056ee47d14b69f0abc0be2`. PR2's binary was tested
directly for CLI SSH; this PR's binary generated the Host configuration
used by native tools.

Each sandbox was stopped before setup. Setup succeeded without starting
it; the subsequent native SSH connection started it automatically. Tests
used a separate SSH configuration file, preserving the user's normal
configuration. Native commands used the generated alias, identity,
host-key settings, and ProxyCommand without supplying transport
arguments manually.

```sh
# CLI SSH: run using the PR2 binary, which has no ssh-setup command.
"$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'printf CLI_E2E_OK'
# Output: CLI_E2E_OK; exit 0.

# Native tools: generate configuration using the PR3 binary.
"$CLI3" sandbox ssh-setup "$SANDBOX_ID" --name my-sandbox \
  --profile "$PROFILE" --ssh-config "$SSH_CONFIG"
ssh -G -F "$SSH_CONFIG" my-sandbox
ssh -F "$SSH_CONFIG" my-sandbox 'printf PR7590_NATIVE_SSH_OK; id -un; exit 23'
# Output includes PR7590_NATIVE_SSH_OK and sandbox-agent; expected exit 23.

ssh -F "$SSH_CONFIG" my-sandbox
# In the interactive shell: print a marker, run id -un and pwd, then exit 29.

scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin
scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin
sftp -F "$SSH_CONFIG" -b - my-sandbox <<EOF_SFTP
put upload.bin /tmp/sftp-upload.bin
get /tmp/sftp-upload.bin sftp-downloaded.bin
EOF_SFTP
```

| Check | Gateway-created sandbox | API-created sandbox |
| --- | --- | --- |
| Setup while stopped; automatic start on connection | Passed | Passed |
| OpenSSH configuration parsing and repeated setup | Passed; files
unchanged on repeated setup | Passed; files unchanged on repeated setup
|
| CLI SSH, native command, interactive shell, PTY, exit status | Passed
| Passed |
| Binary stdin/stdout, native SCP upload/download, native SFTP
upload/download | Passed; exact 1 MiB payload in each case | Passed;
exact 1 MiB payload in each case |
| Preserve unrelated configuration; reject alias overwrite; explicit
overwrite; invalid alias/profile handling | Passed | Passed |
| Delete test sandbox and remove test-owned aliases | Passed | Passed |

Payload: `bytes(range(256)) * 4096`; SHA-256
`fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83`.

The earlier live failures exposed recovery defects in PR2's transport,
now fixed there. A local test proxy forwarded authenticated requests to
a real sandbox while injecting startup failures. The previous build
reproduced both visible failures; the tested revision recovered
automatically, with no caller retries:

- ExecuteCommand unavailable before registration: attach returned
NotFound, then execution retried with the same command ID.
- Lost response after real command registration: the daemon rejected
duplicate execution and the client attached to the original command.
- Initial forwarding HTTP 503: the CLI stopped the failed invocation and
started a fresh server. Native SSH and both SCP directions completed
with exact bytes.
- HTTP 503 after the sandbox had already accepted the WebSocket: cleanup
and a fresh invocation recovered without retrying a handover.
- Setup lasting over one minute followed by HTTP 503: recovery still
completed within the shared startup deadline.

A separate 120-second real session survived a forced network
interruption with the same remote process and all 2,481,600 stream bytes
verified. The original upstream interruption could not be attributed
from the available logs; the client failure modes were reproduced and
fixed.

Local sandbox and SSH suites, race checks, formatting, checks, and
linters pass. Isaac's final local review has no remaining findings. The
installed OpenSSH client was 8.2, so native SCP tested legacy SCP;
native SFTP was tested separately. OpenSSH 9's default-SFTP SCP and
desktop IDEs were not exercised.

### Validation after nonblocking review fixes (2026-11-19)

Tested a clean build of this PR at
`e7e1fc7696145f9a5b87f6b071bf2cd231e092bb` against two newly created
real staging sandboxes, one created through `POST
/api/2.0/lakebox/sandboxes` and one through the public Sandboxes API,
`POST /api/2.0/sandboxes`. Each PR/API combination used its own sandbox;
binaries were checked against their commit versions and SHA-256 before
and after testing.

PR3 was tested using its own binary for CLI SSH, `ssh-setup`, and every
native-tool ProxyCommand. Setup ran while the sandbox was stopped and
did not start it. Repeated setup left both configuration files
unchanged; `ssh -G` parsed the saved profile, identity, host-key checks,
and ProxyCommand. Tests also checked alias overwrite protection and
explicit overwrite while preserving unrelated configuration.

| Check | Gateway-created | Public API-created |
| --- | --- | --- |
| CLI SSH and native SSH automatically start a stopped sandbox | Passed
| Passed |
| Remote stdout, login user, and expected exit 23 | Passed | Passed |
| Interactive CLI and native SSH, typed commands, expected exit 29 |
Passed | Passed |
| Explicit PTY allocation through CLI and native SSH | Passed | Passed |
| 1 MiB binary stdin/stdout through CLI and native SSH | Exact match |
Exact match |
| Native SCP upload/download and native SFTP upload/download | Exact
match | Exact match |
| Delete test sandbox and confirm it is missing; preserve personal SSH
configuration | Passed | Passed |

The PTY test launched each interactive command in a local
pseudo-terminal, waited for the remote shell prompt, typed a unique
marker, `id -un`, `pwd`, and `exit 29`, then checked the output and exit
status. Binary tests used `bytes(range(256)) * 4096` and compared stdout
and downloaded files byte for byte. Payload SHA-256:
`fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83`.

```sh
"$CLI3" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'id -un; exit 23'
"$CLI3" sandbox ssh-setup "$SANDBOX_ID" --name my-sandbox \
  --profile "$PROFILE" --ssh-config "$SSH_CONFIG"
ssh -G -F "$SSH_CONFIG" my-sandbox
ssh -F "$SSH_CONFIG" my-sandbox
ssh -F "$SSH_CONFIG" my-sandbox cat < upload.bin > roundtrip.bin
scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin
scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin
sftp -F "$SSH_CONFIG" -b - my-sandbox <<EOF_SFTP
put upload.bin /tmp/sftp.bin
get /tmp/sftp.bin sftp-downloaded.bin
EOF_SFTP
```

The full sandbox and compute SSH suites, affected race tests,
formatting, checks, lint, and local Isaac review passed.

Commands above show the tested operations with workspace-specific values
and unique remote file names replaced by variables. Installed OpenSSH
8.2 uses legacy SCP; SFTP was exercised separately. The earlier
long-session handover results remain historical; this rerun covers the
nonblocking fixes and both creation APIs.

_This PR was written with Codex._

---------

Co-authored-by: Hsing Tang <HsingTang@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants