Repository navigation
Conversation
The client only allowed GPU_1xA10 and GPU_8xH100, so newer AI Runtime GPU types such as GPU_1xRTX6000 were rejected before the job was submitted even though the server side never looks at the accelerator. Validate the GPU_<count>x<model> shape instead and let the workspace decide which types it offers. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
Thanks for the contribution! This pull request was opened from a fork. If you're a member of the Databricks organization, please request access to the If you're an external contributor, no action is needed; a maintainer will review your pull request. |
Export the accelerator types `databricks air` accepts as KnownGPUTypes and reuse them in the ssh client instead of a shape regex, so the two commands agree on the GPU types they offer. Co-authored-by: Isaac <no-reply@databricks.com>
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
ssh connect --acceleratorssh connect --accelerator
| // KnownGPUTypes lists the accelerator types this CLI accepts, in the order they are shown | ||
| // to users. `ssh connect --accelerator` validates against the same list, so the two entry | ||
| // points to AI Runtime GPUs agree on the types they offer. | ||
| func KnownGPUTypes() []string { |
There was a problem hiding this comment.
can we call AICM's ListComputeOptions to get the list of gpu types available to the user in their workspace? AIR CLI does this already.
There was a problem hiding this comment.
When Jobs provisions the SSH session, AICM’s CreateSession already checks whether the workspace can use the requested GPU. Do we really need the check in CLI using one additional RPC call?
| // which ones a workspace can use comes from the server, not this CLI. | ||
| func parseGPUType(value string) (gpuType, error) { | ||
| switch gpuType(value) { | ||
| case gpuType1xA10, gpuType8xH100, gpuType1xH100, gpuType8xB300, gpuType1xRTX6000: |
There was a problem hiding this comment.
@caroline-db I thought you already removed this list from the CLI client code in favor of the server-side GPU type list. Is this one still used?
| if o.Accelerator != "" && o.Accelerator != "GPU_1xA10" && o.Accelerator != "GPU_8xH100" { | ||
| return fmt.Errorf("invalid accelerator value: %q, expected %q or %q", o.Accelerator, "GPU_1xA10", "GPU_8xH100") | ||
| // The same GPU types `databricks air` accepts; the workspace still decides which it offers. | ||
| if o.Accelerator != "" && !slices.Contains(aircmd.KnownGPUTypes(), o.Accelerator) { |
There was a problem hiding this comment.
The validation is more nuanced: Not all environment versions work with all GPU types. For example, B300 requires v6+. Can you change the validation here to check each environment-version × accelerator combination?
Even better: move the validation to server side (AI Training) and then we can add more GPU types any time without asking the users to upgrade CLI.
Changes
databricks ssh connect --acceleratornow accepts every AI Runtime GPU type the CLI knows, the listdatabricks airuses incmd/air/compute.go:GPU_1xA10,GPU_1xH100,GPU_8xH100,GPU_8xB300andGPU_1xRTX6000. The list is exported asaircmd.KnownGPUTypes()so both commands share one definition, and the flag help and the validation error list the accepted values.Why
The ssh client had its own allowlist of two GPU types, so newer AI Runtime GPUs such as RTX6000 were rejected before the SSH server job was submitted, even though the server never looks at the accelerator and the Jobs API accepts the type. Reusing the
airlist means adding a GPU type in one place covers both commands.Tests
Validatetests gain aGPU_1xRTX6000case and a well-formed but unknownGPU_2xA10case, and the invalid-value message lists the accepted types.go test ./cmd/air/ ./experimental/ssh/...passes;parseGPUTypenow reads the shared list and its existing tests still pass.databricks ssh connect --accelerator=GPU_1xRTX6000 --environment-version=6started the SSH server job on a 1xRTX6000 serverless GPU in a staging workspace and the SSH handshake completed. The unpatched CLI fails the same command withinvalid accelerator value.validate-generatedandtask test-exp-sshfail in their setup step because fork PRs get no OIDC token for the JFrog proxy, before they reach this change.This pull request and its description were written by Isaac.