Skip to content

ssh: default serverless CPU to environment version 5 - #7847

Open
anton-107 wants to merge 2 commits into
mainfrom
ssh-cpu-env-v5
Open

anton-107 wants to merge 2 commits into
mainfrom
ssh-cpu-env-v5

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Changes

databricks ssh connect to serverless CPU compute now submits the SSH server job with environment version 5 when --environment-version is not set. Serverless GPU (--accelerator) stays on 4. An explicit --environment-version still wins, and --base-environment and --cluster are unchanged.

  • DefaultEnvironmentVersion is split into DefaultCPUEnvironmentVersion (5) and DefaultGPUEnvironmentVersion (4). The submit spec, telemetry and the default connection name all resolve an unset version from --accelerator, so the name hashes the version that is actually submitted.
  • Connection names:
    • CPU default names change, so CPU users start a new v5 server on their first connect after upgrading.
    • --environment-version 4 still maps to the old CPU name.
    • GPU default names are unchanged.
  • --environment-version now defaults to 0 (not set), and its help names both defaults. The help example uses 6, since 5 is now the CPU default.

Why

The tunnel stayed on version 4 because the REPL sandbox on CPU environments 5 and later blocked the driver proxy. The backend fix reached all production regions on 2026-11-05. Version 4 blocks IPv4 loopback inside the session, so a Jupyter kernel started over SSH never answers; on 5 it does.

GPU stays on 4 because the standard GPU environments 5 and 6 ship no PyTorch.

Tests

  • Unit tests cover:
    • the submitted version: CPU and GPU, default and explicit, and GPU in proxy mode;
    • pinned default connection names, including unchanged GPU names;
    • telemetry;
    • a ProxyCommand round trip through the ssh connect flags.
  • Acceptance: the only golden change is connect-serverless-cpu submitting "5". connect-serverless-gpu still submits "4".
  • Manual, 2026-11-20: the released v1.27.0 with --environment-version 5 and 6, on production workspaces in Azure westus, AWS us-west-2 and AWS us-east-1.
    • Every CPU session connected.
    • A Jupyter kernel answered in 6 to 9 seconds, the Databricks %pip magic registered, and Spark worked.
    • The version 4 controls' kernels never answered.

This pull request and its description were written by Isaac.

`ssh connect` to serverless CPU compute now submits the SSH server job with
environment version 5 when --environment-version is not set. GPU
(--accelerator) stays on 4, because the standard version 5 and 6 environments
ship no torch on GPU. An explicit --environment-version still wins, and base
environments and dedicated clusters are unchanged.

The single DefaultEnvironmentVersion constant becomes DefaultCPUEnvironmentVersion
and DefaultGPUEnvironmentVersion, resolved from the accelerator by
defaultEnvironmentVersion. The submit spec, telemetry and the default connection
name all go through it, so the name hashes the version that is submitted. CPU
default names change, moving CPU users to a new v5 server; GPU default names are
byte-for-byte unchanged.

The --environment-version flag now defaults to 0 (not set) instead of a single
version, and its help names both defaults. That makes the post-parse reset to 0
unnecessary, and ToProxyCommand keeps persisting the flag only when it was set
explicitly, so a GPU IDE connection still resolves to 4 through --accelerator.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested review from a team as code owners November 20, 2026 15:09
Co-authored-by: Isaac <no-reply@databricks.com>

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 2c12b95. Each finding was verified against the code.

No blocking issues found. 1 non-blocking comment(s) below.

This PR splits the default serverless environment version by compute type: CPU defaults to 5, GPU stays on 4. The flag default moves to 0 (unset) so the actual default is resolved from --accelerator. Connection-name hashing, telemetry, and the submit path all correctly resolve through the new defaultEnvironmentVersion helper. The change is well-tested with pinned connection-name checks and a ProxyCommand round-trip test. One naming nit on the new test file.

@@ -0,0 +1,47 @@
package ssh

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This file declares package ssh (not package ssh_test) to reach the unexported newConnectCommand. The repo convention for same-package tests is to name the file with an _internal_test.go suffix—see client_internal_test.go and submit_internal_test.go one directory over. Rename to connect_internal_test.go to match.

@github-actions

Copy link
Copy Markdown
Contributor

Commit: 2c12b9531 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (8b7e509) latest (v1.27.0)
TestAccept/ssh/connect-serverless-cpu ✅ ❌ ❌
TestAccept/ssh/connect-serverless-cpu: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌

main (8b7e509):

acceptance_test.go:1402: Diff:
        --- ssh/connect-serverless-cpu/output.txt
        +++ /tmp/TestAcceptsshconnect-serverless-cpu3219536239/002/output.txt
        @@ -10,7 +10,7 @@
               {
                 "environment_key": "ssh_tunnel_serverless",
                 "spec": {
        -          "environment_version": "5"
        +          "environment_version": "4"
                 }
               }
             ],
        
    acceptance_test.go:1180: 
        LOG.stderr: Connecting to databricks-cpu-df3cef67...
        LOG.stderr: Uploading binaries...
        LOG.stderr: Starting SSH server...
        LOG.stderr: Job submitted successfully with run ID: 8795187866458163086
        LOG.stderr: Health check successful, starting ssh WebSocket connection...
        LOG.stderr: Connected!
        LOG.stderr: Received termination signal, cleaning up...
FAIL

latest (v1.27.0):

acceptance_test.go:1402: Diff:
        --- ssh/connect-serverless-cpu/output.txt
        +++ /tmp/TestAcceptsshconnect-serverless-cpu1468055493/002/output.txt
        @@ -2,40 +2,3 @@
         === CLI ssh connect (local sshd)
         
         >>> print_requests.py //api/2.2/jobs/runs/submit
        -{
        -  "method": "POST",
        -  "path": "/api/2.2/jobs/runs/submit",
        -  "body": {
        -    "environments": [
        -      {
        -        "environment_key": "ssh_tunnel_serverless",
        -        "spec": {
        -          "environment_version": "5"
        -        }
        -      }
        -    ],
        -    "run_name": "ssh-server-bootstrap-[CPU_CONN]",
        -    "tasks": [
        -      {
        -        "environment_key": "ssh_tunnel_serverless",
        -        "notebook_task": {
        -          "base_parameters": {
        -            "authorizedKeySecretName": "client-public-key",
        -            "keepDetachedProcesses": "false",
        -            "maxClients": "10",
        -            "secretScopeName": "[USERNAME]-[CPU_CONN]-ssh-tunnel-keys",
        -            "serverless": "true",
        -            "sessionId": "[CPU_CONN]",
        -            "shutdownDelay": "10m0s",
        -            "usagePolicyId": "",
        -            "version": "[CLI_VERSION]"
        -          },
        -          "notebook_path": "/Workspace/Users/[USERNAME]/.databricks/ssh-tunnel/[CLI_VERSION]/[CPU_CONN]/ssh-server-bootstrap"
        -        },
        -        "task_key": "start_ssh_server",
        -        "timeout_seconds": 86400
        -      }
        -    ],
        -    "timeout_seconds": 86400
        -  }
        -}
        
    acceptance_test.go:1402: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1402
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1155
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:686
        	Error:      	Not equal: 
        	            	expected: "Connection successful\n"
        	            	actual  : ""
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1 @@
        	            	-Connection successful
        	            	 
        	Test:       	TestAccept/ssh/connect-serverless-cpu
        	Messages:   	ssh/connect-serverless-cpu/out.stdout.txt vs /tmp/TestAcceptsshconnect-serverless-cpu1468055493/001/out.stdout.txt
    acceptance_test.go:1180: 
        LOG.stderr: Connecting to databricks-cpu-b30460a1...
        LOG.stderr: Uploading binaries...
        LOG.stderr: Error: failed to upload ssh-tunnel binaries: failed to get archive for architecture amd64: failed to open /tmp/TestAcceptsshconnect-serverless-cpu1468055493/001/releases/databricks_cli_1.27.0_linux_amd64.zip: open /tmp/TestAcceptsshconnect-serverless-cpu1468055493/001/releases/databricks_cli_1.27.0_linux_amd64.zip: no such file or directory
        LOG.stderr: 
        LOG.stderr: Exit code: 1
FAIL

@anton-107
anton-107 requested a review from rclarey November 20, 2026 15:18
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 5bb9848

Run: 44760778543

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 6:07
💚​ aws-windows-latest-4core-16gb 1 274 18 4:54
💚​ azure linux-2core-8gb 1 271 20 6:46
💚​ azure-windows-latest-4core-16gb 1 273 18 4:03
💚​ gcp linux-2core-8gb 1 272 20 5:59
💚​ gcp-windows-latest-4core-16gb 1 274 18 6:35
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:38 aws-windows-latest-4core-16gb TestAccept
4:01 gcp-windows-latest-4core-16gb TestAccept
3:14 azure-windows-latest-4core-16gb TestAccept
2:14 gcp-windows-latest-4core-16gb TestTelemetryEndpoint
2:05 gcp linux-2core-8gb TestAccept
2:00 azure linux-2core-8gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants