Skip to content

Configure native SSH and SCP with sandbox ssh-setup - #7590

Merged
HsingTang merged 3 commits into
mainfrom
HsingTang/stack/sandbox-ssh-setup
Nov 20, 2026
Merged

HsingTang merged 3 commits into
mainfrom
HsingTang/stack/sandbox-ssh-setup

Conversation

@HsingTang

@HsingTang HsingTang commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR

Use this link to review incremental changes.


Changes

Add databricks sandbox ssh-setup SANDBOX_ID --name ALIAS --profile PROFILE to write a local SSH Host configuration using the existing sandbox proxy mode. Native ssh, scp, and sftp can then connect using that alias. Setup requires an explicit Databricks profile, reuses local keys, retains a selected custom Databricks configuration file, and preserves unrelated SSH configuration. It does not start the sandbox.

Sandbox SSH configuration helpers live alongside the other new sandbox implementation under experimental/ssh/sandbox.

Why

Support native SSH tools through the same authenticated forwarding path as databricks sandbox ssh. Interactive CLI SSH continues to work without a persistent Host entry or an explicit setup step.

Tests

Unit tests cover profile requirements, alias validation, idempotency, overwrite protection, and preservation of unrelated configuration. Installed OpenSSH parses the generated configuration. Acceptance tests exercise setup, repeated setup, saved connection settings, and invalid input.

./task test-sandbox test-exp-ssh, affected race tests, and ./task fmt checks lint pass. The package-layout change was reviewed locally with Isaac, with no blocking findings.

Real-sandbox validation (2026-11-19)

Passed against newly created staging sandboxes from both APIs: POST /api/2.0/lakebox/sandboxes and POST /api/2.0/sandboxes. Tested clean builds of this PR at 88a9af89871bdc738512594c2c95cfb16df6297c and PR2 at 6ab5086fb2f078c21c056ee47d14b69f0abc0be2. PR2's binary was tested directly for CLI SSH; this PR's binary generated the Host configuration used by native tools.

Each sandbox was stopped before setup. Setup succeeded without starting it; the subsequent native SSH connection started it automatically. Tests used a separate SSH configuration file, preserving the user's normal configuration. Native commands used the generated alias, identity, host-key settings, and ProxyCommand without supplying transport arguments manually.

# CLI SSH: run using the PR2 binary, which has no ssh-setup command.
"$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'printf CLI_E2E_OK'
# Output: CLI_E2E_OK; exit 0.

# Native tools: generate configuration using the PR3 binary.
"$CLI3" sandbox ssh-setup "$SANDBOX_ID" --name my-sandbox \
  --profile "$PROFILE" --ssh-config "$SSH_CONFIG"
ssh -G -F "$SSH_CONFIG" my-sandbox
ssh -F "$SSH_CONFIG" my-sandbox 'printf PR7590_NATIVE_SSH_OK; id -un; exit 23'
# Output includes PR7590_NATIVE_SSH_OK and sandbox-agent; expected exit 23.

ssh -F "$SSH_CONFIG" my-sandbox
# In the interactive shell: print a marker, run id -un and pwd, then exit 29.

scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin
scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin
sftp -F "$SSH_CONFIG" -b - my-sandbox <<EOF_SFTP
put upload.bin /tmp/sftp-upload.bin
get /tmp/sftp-upload.bin sftp-downloaded.bin
EOF_SFTP
Check Gateway-created sandbox API-created sandbox
Setup while stopped; automatic start on connection Passed Passed
OpenSSH configuration parsing and repeated setup Passed; files unchanged on repeated setup Passed; files unchanged on repeated setup
CLI SSH, native command, interactive shell, PTY, exit status Passed Passed
Binary stdin/stdout, native SCP upload/download, native SFTP upload/download Passed; exact 1 MiB payload in each case Passed; exact 1 MiB payload in each case
Preserve unrelated configuration; reject alias overwrite; explicit overwrite; invalid alias/profile handling Passed Passed
Delete test sandbox and remove test-owned aliases Passed Passed

Payload: bytes(range(256)) * 4096; SHA-256 fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83.

The earlier live failures exposed recovery defects in PR2's transport, now fixed there. A local test proxy forwarded authenticated requests to a real sandbox while injecting startup failures. The previous build reproduced both visible failures; the tested revision recovered automatically, with no caller retries:

  • ExecuteCommand unavailable before registration: attach returned NotFound, then execution retried with the same command ID.
  • Lost response after real command registration: the daemon rejected duplicate execution and the client attached to the original command.
  • Initial forwarding HTTP 503: the CLI stopped the failed invocation and started a fresh server. Native SSH and both SCP directions completed with exact bytes.
  • HTTP 503 after the sandbox had already accepted the WebSocket: cleanup and a fresh invocation recovered without retrying a handover.
  • Setup lasting over one minute followed by HTTP 503: recovery still completed within the shared startup deadline.

A separate 120-second real session survived a forced network interruption with the same remote process and all 2,481,600 stream bytes verified. The original upstream interruption could not be attributed from the available logs; the client failure modes were reproduced and fixed.

Local sandbox and SSH suites, race checks, formatting, checks, and linters pass. Isaac's final local review has no remaining findings. The installed OpenSSH client was 8.2, so native SCP tested legacy SCP; native SFTP was tested separately. OpenSSH 9's default-SFTP SCP and desktop IDEs were not exercised.

Validation after nonblocking review fixes (2026-11-19)

Tested a clean build of this PR at e7e1fc7696145f9a5b87f6b071bf2cd231e092bb against two newly created real staging sandboxes, one created through POST /api/2.0/lakebox/sandboxes and one through the public Sandboxes API, POST /api/2.0/sandboxes. Each PR/API combination used its own sandbox; binaries were checked against their commit versions and SHA-256 before and after testing.

PR3 was tested using its own binary for CLI SSH, ssh-setup, and every native-tool ProxyCommand. Setup ran while the sandbox was stopped and did not start it. Repeated setup left both configuration files unchanged; ssh -G parsed the saved profile, identity, host-key checks, and ProxyCommand. Tests also checked alias overwrite protection and explicit overwrite while preserving unrelated configuration.

Check Gateway-created Public API-created
CLI SSH and native SSH automatically start a stopped sandbox Passed Passed
Remote stdout, login user, and expected exit 23 Passed Passed
Interactive CLI and native SSH, typed commands, expected exit 29 Passed Passed
Explicit PTY allocation through CLI and native SSH Passed Passed
1 MiB binary stdin/stdout through CLI and native SSH Exact match Exact match
Native SCP upload/download and native SFTP upload/download Exact match Exact match
Delete test sandbox and confirm it is missing; preserve personal SSH configuration Passed Passed

The PTY test launched each interactive command in a local pseudo-terminal, waited for the remote shell prompt, typed a unique marker, id -un, pwd, and exit 29, then checked the output and exit status. Binary tests used bytes(range(256)) * 4096 and compared stdout and downloaded files byte for byte. Payload SHA-256: fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83.

"$CLI3" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'id -un; exit 23'
"$CLI3" sandbox ssh-setup "$SANDBOX_ID" --name my-sandbox \
  --profile "$PROFILE" --ssh-config "$SSH_CONFIG"
ssh -G -F "$SSH_CONFIG" my-sandbox
ssh -F "$SSH_CONFIG" my-sandbox
ssh -F "$SSH_CONFIG" my-sandbox cat < upload.bin > roundtrip.bin
scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin
scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin
sftp -F "$SSH_CONFIG" -b - my-sandbox <<EOF_SFTP
put upload.bin /tmp/sftp.bin
get /tmp/sftp.bin sftp-downloaded.bin
EOF_SFTP

The full sandbox and compute SSH suites, affected race tests, formatting, checks, lint, and local Isaac review passed.

Commands above show the tested operations with workspace-specific values and unique remote file names replaced by variables. Installed OpenSSH 8.2 uses legacy SCP; SFTP was exercised separately. The earlier long-session handover results remain historical; this rerun covers the nonblocking fixes and both creation APIs.

This PR was written with Codex.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Commit: d8d0d7266 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (7f25e1e) latest (v1.27.0)
TestAccept/cmd/sandbox/ssh-setup ✅ ❌ ❌
TestAccept/cmd/sandbox/ssh-setup: branch ✅ | main (7f25e1e) ❌ | latest (v1.27.0) ❌

main (7f25e1e):

acceptance_test.go:1400: Diff:
        --- cmd/sandbox/ssh-setup/output.txt
        +++ /tmp/TestAcceptcmdsandboxssh-setup274610017/002/output.txt
        @@ -1,25 +1,35 @@
         
         >>> [CLI] sandbox ssh-setup test --name my-sandbox --profile sandbox-test --releases-dir ./releases
        -Created SSH configuration at [TEST_TMP_DIR]/.databricks/ssh-tunnel-configs/my-sandbox. Connect with 'ssh my-sandbox' or 'scp ./file my-sandbox:/tmp/file'.
        +Error: unknown flag: --name
         
        ->>> [CLI] sandbox ssh-setup test --name my-sandbox --profile sandbox-test --releases-dir ./releases
        -Created SSH configuration at [TEST_TMP_DIR]/.databricks/ssh-tunnel-configs/my-sandbox. Connect with 'ssh my-sandbox' or 'scp ./file my-sandbox:/tmp/file'.
        +Usage:
        +  databricks sandbox [command]
         
        ->>> cat [TEST_TMP_DIR]/.ssh/config
        -Include "[TEST_TMP_DIR]/.databricks/ssh-tunnel-configs/*"
        +Available Commands:
        +  config      Configure a Sandbox's name and auto-stop policy
        +  create      Create a new Sandbox environment
        +  default     Set the default Sandbox for SSH
        +  delete      Delete a Sandbox environment
        +  forward     Forward local ports to apps in a Sandbox environment
        +  fs          Filesystem related commands
        +  list        List your Sandbox environments
        +  register    Register this machine for sandbox SSH access
        +  ssh         SSH into a Sandbox environment
        +  ssh-key     Manage SSH keys registered with the sandbox service
        +  start       Start a stopped Sandbox environment
        +  status      Show Sandbox environment status
        +  stop        Stop a running Sandbox environment
         
        ->>> cat [TEST_TMP_DIR]/.databricks/ssh-tunnel-configs/my-sandbox
        +Flags:
        +  -h, --help[=true|false]   help for sandbox
         
        -Host my-sandbox
        -    User sandbox-agent
        -    ConnectTimeout 900
        -    ServerAliveInterval 30
        -    StrictHostKeyChecking yes
        -    UserKnownHostsFile "[TEST_TMP_DIR]/.databricks/ssh-tunnel-known-hosts/[SSH_IDENTITY]"
        -    HostKeyAlias [SSH_IDENTITY]
        -    IdentitiesOnly yes
        -    IdentityFile "[TEST_TMP_DIR]/.databricks/ssh-tunnel-keys/[SSH_IDENTITY]/id_ed25519"
        -    ProxyCommand [CLI] sandbox ssh sandboxes/test --proxy --profile sandbox-test --releases-dir [TEST_TMP_DIR]/releases --ssh-identity [SSH_IDENTITY] --ssh-config-file [TEST_TMP_DIR]/databrickscfg
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
         
        ->>> musterr [CLI] sandbox ssh-setup test --name ../escape
        -Error: SSH host alias must contain only letters, numbers, dots, underscores or hyphens
        +Use "databricks sandbox [command] --help" for more information about a command.
        +
        +
        +Exit code: 1
        
FAIL

latest (v1.27.0): same output as main (7f25e1e)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: d8d0d72

Run: 44653772492

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 4:28
💚​ aws-windows-latest-4core-16gb 1 274 18 3:45
💚​ azure linux-2core-8gb 1 271 20 4:04
💚​ azure-windows-latest-4core-16gb 1 273 18 3:39
💚​ gcp linux-2core-8gb 1 272 20 4:27
💚​ gcp-windows-latest-4core-16gb 1 274 18 3:49
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:47 gcp-windows-latest-4core-16gb TestAccept
3:43 aws-windows-latest-4core-16gb TestAccept
3:37 azure-windows-latest-4core-16gb TestAccept

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-setup branch from b9424a8 to c26a360 Compare November 13, 2026 21:43
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from 417c077 to 5441e69 Compare November 13, 2026 21:43
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-setup branch from c26a360 to a725f0c Compare November 13, 2026 22:12
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch 2 times, most recently from 9434fda to 637d42e Compare November 18, 2026 05:45
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-setup branch from a725f0c to 1a2cf4d Compare November 18, 2026 05:45
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from 637d42e to c4637bc Compare November 18, 2026 06:01
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-setup branch 2 times, most recently from b69ba16 to acd94d2 Compare November 19, 2026 05:15
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from c4637bc to aab1071 Compare November 19, 2026 05:15
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-setup branch 2 times, most recently from 987588f to 4d9dfba Compare November 19, 2026 16:18
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from 69946b9 to 12e4dd1 Compare November 19, 2026 16:18

@anton-107 anton-107 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. go test -race passes for experimental/ssh/sandbox, internal/sshconfig and cmd/sandbox, and TestAccept/cmd/sandbox/ssh-setup passes locally. I also had real OpenSSH (ssh -G) parse a generated config with a HOME containing a space and %d, a CLI path with a space, a profile named p 1, and a user ControlPath ~/.ssh/cm-%r@%h:%p. The %% escaping is correct, and the short alias keeps ControlPath within the length limit.

Non-blocking: cmd/sandbox/README.md still says a broken forwarding connection ends the SSH session. Since #7589 negotiates resume v2, that paragraph should describe resume instead.

LGTM.

@HsingTang
HsingTang marked this pull request as ready for review November 19, 2026 18:31
@HsingTang
HsingTang requested review from a team as code owners November 19, 2026 18:31

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 4d9dfba. Each finding was verified against the code.

No blocking issues found. 2 non-blocking comment(s) below.

This PR adds databricks sandbox ssh-setup to write a persistent SSH Host entry for native ssh/scp/sftp through the existing sandbox proxy. The implementation is clean: alias validation is thorough, idempotency and overwrite protection work correctly, and the acceptance and unit tests cover the main paths well. Two minor items below.

Comment thread experimental/ssh/sandbox/config.go Resolved
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-setup branch 2 times, most recently from 88a9af8 to e7e1fc7 Compare November 19, 2026 22:45
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from 6ab5086 to c2d75cd Compare November 19, 2026 22:45
@HsingTang

Copy link
Copy Markdown
Contributor Author

@anton-107 Updated the README in e7e1fc7 to describe 25-minute stream renewal, resuming the same session for up to 60 seconds after an unexpected disconnect, and termination if recovery fails or the remote session is gone.

This PR's own binary passed CLI SSH and ssh-setup/native SSH/SCP/SFTP against separate freshly created Gateway and public API sandboxes. Setup while stopped, idempotency, configuration parsing, overwrite protection, interactive sessions, PTY, exit status, and exact 1 MiB transfers passed. The test plan records the commands and results; local checks and Isaac review also pass.

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-port-forward branch from c2d75cd to 49d6175 Compare November 20, 2026 00:33
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-setup branch from 747b807 to 7af91ee Compare November 20, 2026 00:33
Base automatically changed from HsingTang/stack/sandbox-ssh-port-forward to main November 20, 2026 01:10
@HsingTang
HsingTang requested a review from a team as a code owner November 20, 2026 01:10
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-ssh-setup branch from 7af91ee to d8d0d72 Compare November 20, 2026 01:26
@HsingTang
HsingTang enabled auto-merge November 20, 2026 01:28
@HsingTang
HsingTang added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 1ede87e Nov 20, 2026
32 checks passed
@HsingTang
HsingTang deleted the HsingTang/stack/sandbox-ssh-setup branch November 20, 2026 01:58
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: 1ede87e

Run: 44657554969

Env 🟨​KNOWN 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 27 1619 1420 159:09
🟨​ azure linux-2core-8gb 4 23 1599 1428 151:02
💚​ gcp linux-2core-8gb 27 1606 1426 168:52
27 interesting tests: 23 RECOVERED, 4 KNOWN
Test Name aws linux-2core-8gb azure linux-2core-8gb gcp linux-2core-8gb
🟨​ TestAccept 💚​R 🟨​K 💚​R
🟨​ TestAccept/bundle/deployment/bind/alert 💚​R 🟨​K 💚​R
🟨​ TestAccept/bundle/deployment/bind/alert/DMS= 💚​R 🟨​K 💚​R
🟨​ TestAccept/bundle/deployment/bind/alert/DMS=true 💚​R 🟨​K 💚​R
💚​ TestAccept/bundle/generate/alert 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/generate/alert/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/generate/alert/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/basic 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/basic/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/basic/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS=true 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
12:50 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
11:00 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
10:44 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:29 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:03 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:35 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:13 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:56 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:16 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:05 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:55 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:42 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:15 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:05 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=
6:05 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:48 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:43 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
5:40 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:09 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
4:57 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
4:55 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
4:46 azure linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:40 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
4:31 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
4:24 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
4:19 gcp linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:18 aws linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:06 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
3:54 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
3:42 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
3:36 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
3:31 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
3:31 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
3:31 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
3:29 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
3:26 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
3:26 gcp linux-2core-8gb TestAccept/bundle/quiet-levels/DMS=
3:25 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
3:24 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
3:14 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
3:13 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:06 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
3:03 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:02 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
3:02 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
2:56 aws linux-2core-8gb TestAccept/bundle/resources/apps/inline_config/DMS=
2:56 gcp linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
2:53 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
2:53 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
2:52 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=

idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
## 🥞 Stacked PR
Use this [link](https://github.com/databricks/cli/pull/7588/files) to
review incremental changes.
-
[**stack/sandbox-websocket-server**](databricks#7588)
[[Files changed](https://github.com/databricks/cli/pull/7588/files)] ←
_this PR_
-
[stack/sandbox-ssh-port-forward](databricks#7589)
[[Files changed](https://github.com/databricks/cli/pull/7589/files)]
- [stack/sandbox-ssh-setup](databricks#7590)
[[Files changed](https://github.com/databricks/cli/pull/7590/files)]

---------
## What did you change, and why?

Sandbox SSH needs a server that works through HTTP/WebSocket port
forwarding without compute initialization. Add the hidden, internal
`databricks sandbox server` command while leaving the existing SSH
command unchanged in this PR.

Keep the implementation under `experimental/ssh/sandbox` with sandbox
CODEOWNERS. Require an invocation token, validate forwarding ports, and
allow one SSH session with the existing handover and versioned resume
protocols. The shared proxy owns session admission and reports
completion or process-start failure; invoking the process factory
consumes the single session, while failed WebSocket upgrades can retry.
The sandbox owns its first-connection deadline and shutdown. Preserve
compute timeout defaults and remove unused proxy options.

## How do you know it works?

Sandbox and compute SSH suites, race tests, acceptance tests, and
`./task fmt checks lint` pass. Tests cover repeated WebSocket
interruptions without losing bytes or restarting the SSH process,
handover through a proxy with a connection lifetime limit, failed
upgrades and process starts, startup deadlines, and cleanup. The SIGTERM
acceptance test verifies a child that remains alive when stdin closes. A
standalone native OpenSSH check built from this PR verifies that both
SSH server processes exit when the server receives SIGTERM.

The dependent client was also validated against a real sandbox,
including SSH, SCP/SFTP, and a 28-minute session with forced-disconnect
recovery and scheduled handover. Local Isaac review findings were
addressed or checked against native OpenSSH behavior. The latest
startup-failure correction passed repeated race tests with 16 concurrent
dials, the full SSH suite, and a fresh local Isaac review. Its dependent
client also passed fresh real-sandbox SSH/SCP/SFTP, stopped-sandbox
startup, eight-second outage recovery, and failure-log retention checks.

### Validation after nonblocking review fixes (2026-11-19)

At `cd5c2165022ec908f438fdf9085d2b5a3636f0a0`, the PID stub publishes
its completed file using a same-directory rename, and the concurrent
startup test waits for all HTTP handlers before checking completion
counts. The fixed test passed 3,000 repetitions with Go's race detector.
The server acceptance script passed 1,000 runs with 16 concurrent
workers. Formatting, lint, and local Isaac review passed with no
remaining reported findings.

_This PR was written with Codex._

---------

Co-authored-by: Hsing Tang <HsingTang@users.noreply.github.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…atabricks#7589)

## 🥞 Stacked PR
Use this [link](https://github.com/databricks/cli/pull/7589/files) to
review incremental changes.
-
[stack/sandbox-websocket-server](databricks#7588)
[[Files changed](https://github.com/databricks/cli/pull/7588/files)]
[MERGED]
-
[**stack/sandbox-ssh-port-forward**](databricks#7589)
[[Files changed](https://github.com/databricks/cli/pull/7589/files)] ←
_this PR_
- [stack/sandbox-ssh-setup](databricks#7590)
[[Files changed](https://github.com/databricks/cli/pull/7590/files)]

---------
## What did you change, and why?

Keep `databricks sandbox ssh` while removing its dependence on the
Gateway SSH listener and key registry. Use reusable local keys, command
execution for expiring public-key entries, and authenticated WebSocket
forwarding through `/sandbox-proxy/`.

The internal proxy prepares and starts the hidden sandbox server, pins
its host key, authenticates with its invocation token, and cleans up its
command. Cache the Linux executable by architecture and version (content
for development builds), retry uploads with a seekable body, and install
uploads atomically. Retain at most eight unleased cached releases and
expire unleased releases after 30 days. A hard link in each session
directory protects its executable during startup and use. Serialize
abandoned-session cleanup, and retain unexpected-failure logs while
cleaning up successful or cancelled sessions.

Renew WebSocket and command-output streams every 25 minutes. Negotiate
the existing resume protocol through server readiness and reauthenticate
each replacement WebSocket. Reattach transiently interrupted command
output using the same command ID and sequence number, with a 60-second
recovery budget that accommodates 30-second server heartbeats. Report
startup progress on stderr and disable inherited OpenSSH connection
multiplexing. Preserve writable inherited command-output pipes by
duplicating their descriptors. Keep startup diagnostics visible and
direct subsequent server logs to a session file.

Keep the implementation under `experimental/ssh/sandbox` and reuse
existing SSH helpers at their current paths. This requires workspace
sandbox HTTP/WebSocket forwarding.

## How do you know it works?

Validated build `aab1071a423c` against a real staging sandbox. After the
review fixes, repeated the SSH and file-transfer checks below on
`69946b9e5164` in a freshly created staging sandbox; all passed.

Latest review revision `12e4dd1c38e6` passed the SSH and file-transfer
operations below in another freshly created staging sandbox. It also
passed:

- Stop the sandbox through the API, then reconnect with `databricks
sandbox ssh`: the CLI reports startup progress and the remote command
succeeds.
- Interrupt this invocation’s connections and block replacement
connections for **eight seconds**: a **120-second** SSH session
preserves its remote process and verifies **2,481,600 exact bytes**,
then exits **0**.
- Force the remote invocation server to exit unexpectedly: the CLI
reports the retained session-directory path, and command execution
confirms that its diagnostic log remains readable.

The validation sandbox was deleted afterward. Local coverage includes 16
concurrent failed-start attempts, a complete SDK upload replay after
HTTP 503, cache pruning with active and starting sessions, concurrent
session cleanup under umask `002`, a long inherited OpenSSH
`ControlPath`, 60-second output retry exhaustion, a silent first 30
seconds, and healthy output after 61 seconds. The new Python test
harnesses also pass with system Python 3.6.

**SSH and file transfer**

Invoked the rebuilt CLI:

```bash
"$CLI" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- true
```

Temporarily placed an `ssh` wrapper on `PATH` to record the arguments
generated by the CLI, then execute system OpenSSH. Reused those same
arguments—including the client identity, pinned host-key settings, and
CLI `ProxyCommand`—for native SSH, SCP, and SFTP.

The connection settings were equivalent to:

```bash
SSH_OPTIONS=(
  -i "$KEY_FILE"
  -o IdentitiesOnly=yes
  -o PreferredAuthentications=publickey
  -o ConnectTimeout=900
  -o StrictHostKeyChecking=yes
  -o "UserKnownHostsFile=$KNOWN_HOSTS_FILE"
  -o "HostKeyAlias=$SSH_IDENTITY"
  -o ServerAliveInterval=30
  -o "ProxyCommand=$CLI sandbox ssh sandboxes/$SANDBOX_ID --proxy --profile $PROFILE --ssh-identity $SSH_IDENTITY"
)
DESTINATION="sandbox-agent@$SSH_IDENTITY"
```

Paths and identity values above represent the actual CLI-generated
values captured during the test.

Ran the following operations, shown as shell equivalents of the Python
subprocess calls:

```bash
# Verify remote stdout and exit-status propagation.
ssh "${SSH_OPTIONS[@]}" "$DESTINATION" \
  'printf NATIVE_SSH_OK; exit 23'

# Verify remote terminal allocation.
ssh "${SSH_OPTIONS[@]}" -tt "$DESTINATION" \
  'test -t 0 && printf PTY_OK'

# Round-trip a binary payload through stdin/stdout.
ssh "${SSH_OPTIONS[@]}" "$DESTINATION" cat \
  < upload.bin > roundtrip.bin

# Upload and download using native SCP.
scp "${SSH_OPTIONS[@]}" upload.bin \
  "$DESTINATION:/home/sandbox-agent/pr7589-live.bin"

scp "${SSH_OPTIONS[@]}" \
  "$DESTINATION:/home/sandbox-agent/pr7589-live.bin" downloaded.bin

# Upload and download using native SFTP.
sftp -b - "${SSH_OPTIONS[@]}" "$DESTINATION" <<EOF
put upload.bin /home/sandbox-agent/pr7589-sftp.bin
get /home/sandbox-agent/pr7589-sftp.bin sftp-downloaded.bin
EOF
```

The payload was `bytes(range(256)) * 4096`—1 MiB containing every byte
value. Python compared the round-trip stdout and both downloaded files
byte for byte with the original.

| Check | Recorded result |
|---|---|
| Remote command | `NATIVE_SSH_OK`; expected exit code **23** |
| PTY | `PTY_OK`; exit **0** |
| Binary stdin/stdout | Exact match |
| SCP upload/download | Exact match |
| SFTP upload/download | Exact match |
| Remove test files | Exit **0** |

Payload SHA-256:

```text
fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83
```

**Connection recovery and handover**

An earlier build, `07981d228a81`, completed a 28-minute real-sandbox
session. A local CONNECT proxy interrupted the actual connections and
blocked replacement connections for two seconds. The verifier checked
every output frame and confirmed unchanged remote process identity.

```json
{
  "exit_code": 0,
  "elapsed_seconds": 1684.558,
  "verified_bytes": 34742400,
  "exact_bytes": true,
  "same_remote_pid": true,
  "fault_resume_verified": true,
  "planned_handover_verified": true
}
```

Server logs confirmed two reattachments and the scheduled 25-minute
handover.

After the final startup-error fix, repeated forced-disconnect recovery
on build `aab1071a423c`: **120 seconds, 2,481,600 verified bytes,
unchanged remote process, successful reattachment, exit 0**. Repeated
the SSH and file-transfer checks above on that build. The 25-minute
handover was not rerun after that final correction.

**Local checks**

Formatting, checks, lint, sandbox/compute SSH tests, affected race
tests, and Windows amd64/Darwin arm64/Linux arm64 cross-compilation
passed.

### Startup recovery validation (2026-11-19)

Tested this PR at `6ab5086fb2f078c21c056ee47d14b69f0abc0be2`. Its CLI
SSH command passed against real sandboxes created through both the
Gateway API and the new Sandboxes API. The matching PR3 build,
`88a9af89871bdc738512594c2c95cfb16df6297c`, supplied Host configuration
for native SSH/SCP/SFTP validation of the same transport.

Both sandbox types passed stopped-sandbox startup, interactive native
SSH, PTY, remote command/exit status, exact binary stdin/stdout, and
native SCP/SFTP upload/download with a 1 MiB payload. Setup while
stopped, repeated setup, overwrite protection, and cleanup also passed.
Native connections used the generated configuration without hand-written
identity or ProxyCommand options.

```sh
"$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'printf CLI_E2E_OK'
# Output: CLI_E2E_OK; exit 0.

# After generating the native-tool configuration with the matching PR3 build:
ssh -F "$SSH_CONFIG" my-sandbox
scp -F "$SSH_CONFIG" upload.bin my-sandbox:/tmp/upload.bin
scp -F "$SSH_CONFIG" my-sandbox:/tmp/upload.bin downloaded.bin
```

Recovery tests used a local proxy connected to a real staging sandbox.
Each native command was invoked once; recovery happened inside the CLI.

| Injected failure | Previous build | Fixed transport |
| --- | --- | --- |
| ExecuteCommand unavailable before command registration | AttachCommand
NotFound ended SSH | Retry with the same command ID; SSH and native SCP
succeeded |
| Response lost after real command registration | Not used as a negative
control | Daemon rejected duplicate execution; attachment recovered the
original command |
| Initial forwarding HTTP 503 | Bad handshake ended SSH | Confirm
termination, then use a fresh server; SSH and both SCP directions passed
|
| HTTP 503 after the remote WebSocket was accepted | Not used as a
negative control | Retired the old invocation and connected through a
fresh server |
| More than one minute of setup before HTTP 503 | Not used as a negative
control | Retried within the shared startup deadline |

A separate 120-second real-session interruption test retained the remote
process and verified all 2,481,600 bytes. The earlier upstream
interruption itself could not be attributed from the available logs;
these client recovery failures were reproduced and fixed.

Regression tests cover safe same-ID execution recovery, refusal to
restart a confirmed missing command, HTTP 429/503 recovery, immediate
HTTP 403 failure, cancellation/cleanup, refusal to retry when server
termination fails, and preserving stdin before connection establishment.
Full sandbox/SSH suites, race tests, formatting/checks/linters, and
local Isaac review pass. Native SCP used installed OpenSSH 8.2; SFTP was
tested separately.

### Validation after nonblocking review fixes (2026-11-19)

Tested a clean build of this PR at
`c2d75cdea22149c44249e6ef1340058ab597da6c` against two newly created
real staging sandboxes, one created through `POST
/api/2.0/lakebox/sandboxes` and one through the public Sandboxes API,
`POST /api/2.0/sandboxes`. Each PR/API combination used its own sandbox;
binaries were checked against their commit versions and SHA-256 before
and after testing.

PR2 was tested independently, without `ssh-setup` or a PR3 binary. A
temporary local `ssh` wrapper recorded the CLI-generated identity,
host-key options, destination, and ProxyCommand, then executed system
OpenSSH. Native SSH/SCP/SFTP reused those captured settings. The wrapper
was used only to capture arguments; it did not replace the transport.

| Check | Gateway-created | Public API-created |
| --- | --- | --- |
| CLI SSH and native SSH automatically start a stopped sandbox | Passed
| Passed |
| Remote stdout, login user, and expected exit 23 | Passed | Passed |
| Interactive CLI and native SSH, typed commands, expected exit 29 |
Passed | Passed |
| Explicit PTY allocation through CLI and native SSH | Passed | Passed |
| 1 MiB binary stdin/stdout through CLI and native SSH | Exact match |
Exact match |
| Native SCP upload/download and native SFTP upload/download | Exact
match | Exact match |
| Delete test sandbox and confirm it is missing; preserve personal SSH
configuration | Passed | Passed |

The PTY test launched each interactive command in a local
pseudo-terminal, waited for the remote shell prompt, typed a unique
marker, `id -un`, `pwd`, and `exit 29`, then checked the output and exit
status. Binary tests used `bytes(range(256)) * 4096` and compared stdout
and downloaded files byte for byte. Payload SHA-256:
`fbbab289f7f94b25736c58be46a994c441fd02552cc6022352e3d86d2fab7c83`.

Timeout cleanup coverage confirms that startup deadlines and readiness
failures retain session diagnostics, while success and explicit user
cancellation remove the session directory. The full sandbox and compute
SSH suites, affected race tests, formatting, checks, lint, and local
Isaac review passed.

```sh
# Direct CLI SSH; the test verified the expected remote exit status.
"$CLI2" sandbox ssh "$SANDBOX_ID" --profile "$PROFILE" -- 'id -un; exit 23'
# For native tools, SSH_OPTIONS and DESTINATION below are captured from this
# PR's own OpenSSH invocation, including its identity and ProxyCommand.
ssh "${SSH_OPTIONS[@]}" "$DESTINATION" cat < upload.bin > roundtrip.bin
scp "${SSH_OPTIONS[@]}" upload.bin "$DESTINATION:/tmp/upload.bin"
scp "${SSH_OPTIONS[@]}" "$DESTINATION:/tmp/upload.bin" downloaded.bin
sftp "${SSH_OPTIONS[@]}" -b - "$DESTINATION" <<EOF_SFTP
put upload.bin /tmp/sftp.bin
get /tmp/sftp.bin sftp-downloaded.bin
EOF_SFTP
```

Commands above show the tested operations with workspace-specific values
and unique remote file names replaced by variables. Installed OpenSSH
8.2 uses legacy SCP; SFTP was exercised separately. The earlier
long-session handover results remain historical; this rerun covers the
nonblocking fixes and both creation APIs.

_This PR was written with Codex._

---------

Co-authored-by: Hsing Tang <HsingTang@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants