Repository navigation
Support accelerator_pool_id in AIR workload configuration - #7698
Conversation
| @@ -0,0 +1 @@ | |||
| * Add `compute.accelerator_pool_id` for GPU Pool selection in AIR workload YAML, retaining `compute.pool_id` as a compatibility alias. ([#7698](https://github.com/databricks/cli/pull/7698)) | |||
There was a problem hiding this comment.
Blocking on Tuesday deployment
|
Commit: Regression test reportSelected 5 changed tests (limit=20, 0 not selected) 5 tests: regression=5 unreleased=0 coverage=0 failing=0
TestAccept/air/config-help: branch ✅ | main (b8046e0) ❌ | latest (v1.27.0) ❌main (b8046e0): latest (v1.27.0): same output as main (b8046e0) TestAccept/air/convert-to-dabs: branch ✅ | main (b8046e0) ❌ | latest (v1.27.0) ❌main (b8046e0): latest (v1.27.0): same output as main (b8046e0) TestAccept/air/help: branch ✅ | main (b8046e0) ❌ | latest (v1.27.0) ❌main (b8046e0): latest (v1.27.0): TestAccept/air/run: branch ✅ | main (b8046e0) ❌ | latest (v1.27.0) ❌main (b8046e0): latest (v1.27.0): same output as main (b8046e0) TestAccept/air/run-submit-deps: branch ✅ | main (b8046e0) ❌ | latest (v1.27.0) ❌main (b8046e0): latest (v1.27.0): same output as main (b8046e0) |
✅ Integration test reportCommit: 894df84
Top 18 slowest tests (at least 2 minutes):
|
| @@ -0,0 +1 @@ | |||
| * Rename `compute.pool_id` to `compute.accelerator_pool_id` for GPU Pool selection in AIR workload YAML. ([#7698](https://github.com/databricks/cli/pull/7698)) | |||
There was a problem hiding this comment.
Is the new name accelerator_pool_id finalized? Since it's a user-facing YAML key that we'd have to support long-term, I'd like to confirm it before it ships.
If it is, this is a breaking change for released users: yaml:"pool_id" shipped in v1.26.0, and after this PR it fails with unknown field. The entry only says "Rename" and doesn't mention HIGH. Could we make it explicit, e.g.:
Breaking:
compute.pool_idis no longer accepted; usecompute.accelerator_pool_id. Also addspriority_class: HIGH.
There was a problem hiding this comment.
Yes this is the finalized name users will use see https://docs.google.com/document/d/12ICYBiWvRFBnteia21DcWt1oZzreokvCJ2RGbVKOEt4/edit?tab=t.0#heading=h.wz3kd1g0z1od
| PriorityClass *string `yaml:"priority_class" help:"Scheduling priority within the pool: BEST_EFFORT (lowest, preemptable) or NORMAL. Requires pool_id." group:"Provisioned capacity (optional)"` | ||
| NumAccelerators int `yaml:"num_accelerators" help:"Total number of GPUs to allocate. Must be a positive multiple of the accelerator type's per-node GPU count. See Docs below for GPUs per node." required:"yes" docs:"/machine-learning/ai-runtime/connecting/cli/yaml-config#accelerator-types"` | ||
| AcceleratorType string `yaml:"accelerator_type" help:"Which accelerator to run on, e.g. GPU_1xA10. Matched case-sensitively." required:"yes" docs:"/machine-learning/ai-runtime/connecting/cli/yaml-config#accelerator-types"` | ||
| AcceleratorPoolID *string `yaml:"accelerator_pool_id" help:"GPU Pool to run on, by ID. Must be 1-255 characters. List available pools with 'databricks air pools list'. An invalid or inaccessible pool is rejected. When omitted, workspace capacity selection applies." group:"GPU Pools (optional)"` |
There was a problem hiding this comment.
Users who still have pool_id in their YAML will get unknown field "compute.pool_id" with no pointer to the new name. closestConfigField (runconfig_load.go:178) only suggests within len/3 edits, and pool_id is 12 edits from accelerator_pool_id, so no "did you mean" fires. Could we add an explicit hint for the renamed key (pool_id -> accelerator_pool_id) and cover it in TestLoadRunConfigAcceleratorPool? Keeping the rejection itself is fine.
There was a problem hiding this comment.
We shouldn't reference pool_id, no one is using it. There's nothing legacy to do here. I am going to push back on this.
There was a problem hiding this comment.
SG. Can you do a follow up PR to deprecate pool_id in YAML? (update all docs / examples /. skills, give warning for a few versions, then finally remove it)
| const ( | ||
| priorityClassBestEffort priorityClass = "BEST_EFFORT" | ||
| priorityClassNormal priorityClass = "NORMAL" | ||
| priorityClassHigh priorityClass = "HIGH" |
There was a problem hiding this comment.
HIGH isn't in the contract this repo generates: bundle/schema/jsonschema.json (jobs.AiRuntimeTaskPriorityClass) and python/databricks/bundles/jobs/_models/ai_runtime_task_priority_class.py both list only BEST_EFFORT and NORMAL, while the comment above says these values match the AiRuntimeTask.PriorityClass proto enum. Could we confirm HIGH against the proto / AICM validator before merge, and regenerate the schema if it's stale?
There was a problem hiding this comment.
Sg, HIGH has been landed for a bit so it seems that we need to regenerate the schema
| } | ||
| if cfg.Compute.PoolID != nil { | ||
| compute["provisioned_capacity_id"] = nv(*cfg.Compute.PoolID, 3) | ||
| // Bundles still use the legacy field supported by their current schema. |
There was a problem hiding this comment.
Related to the HIGH question: priority_class passes straight through into the bundle's ai_runtime_task (line 338), so with HIGH the generated bundle YAML would carry a value the bundle schema enum rejects. The DABs test only covers NORMAL (convert_to_dabs_test.go:197-200). Could we add a HIGH case, or note the gap if the bundle schema is expected to change alongside?
There was a problem hiding this comment.
Sure I'll add a test
| title "new submission fields are documented" | ||
| trace $CLI air run -h config.mlflow_artifact_location | ||
| trace $CLI air run -h config.compute.pool_id | ||
| trace $CLI air run -h config.compute.accelerator_pool_id |
There was a problem hiding this comment.
The pool_id rejection, an invalid priority_class, and priority_class without a pool are only covered by unit tests. Could we add trace $CLI air run -h config.compute.pool_id here (after the rename hint above lands), so the migration path is pinned in acceptance output?
There was a problem hiding this comment.
Sure, updated
| PriorityClass *string `yaml:"priority_class" help:"Scheduling priority within the pool: BEST_EFFORT (lowest, preemptable) or NORMAL. Requires pool_id." group:"Provisioned capacity (optional)"` | ||
| NumAccelerators int `yaml:"num_accelerators" help:"Total number of GPUs to allocate. Must be a positive multiple of the accelerator type's per-node GPU count. See Docs below for GPUs per node." required:"yes" docs:"/machine-learning/ai-runtime/connecting/cli/yaml-config#accelerator-types"` | ||
| AcceleratorType string `yaml:"accelerator_type" help:"Which accelerator to run on, e.g. GPU_1xA10. Matched case-sensitively." required:"yes" docs:"/machine-learning/ai-runtime/connecting/cli/yaml-config#accelerator-types"` | ||
| AcceleratorPoolID *string `yaml:"accelerator_pool_id" help:"GPU Pool to run on, by ID. Must be 1-255 characters. List available pools with 'databricks air pools list'. An invalid or inaccessible pool is rejected. When omitted, workspace capacity selection applies." group:"GPU Pools (optional)"` |
There was a problem hiding this comment.
nit: "GPU Pool" here vs "GPU pool" in pools.go:23; could we pick one? Also, "An invalid or inaccessible pool is rejected" and "When omitted, workspace capacity selection applies" describe backend behavior we can't verify from the CLI. Worth confirming with the backend team or softening the wording.
There was a problem hiding this comment.
we can capitalize pool, we haven't finalized the backend behavior yet so it's kept intentionally vague
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
Requesting changes to back maggiewang-db's open threads. Two of them should block the merge: compute.pool_id (shipped in v1.18.0 through v1.26.0) now fails with no hint toward accelerator_pool_id, and HIGH passes air run validation but produces a convert-to-dabs bundle that the bundle schema's jobs.AiRuntimeTaskPriorityClass enum rejects.
I'm going to push back strongly here. pool_id is not used by anyone and should not be supported. Wrt HIGH it seems that the proto needs to be regenerated. If there's some lag I'll leave it out of this pr. This is to support accelerator_pool_id |
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
thanks for dropping HIGH. I'd still like a pointer for pool_id. It shipped in v1.18.0 through v1.26.0, so anyone who used it gets unknown field "compute.pool_id" and nothing naming accelerator_pool_id.
suggested fix: special-case pool_id in the unknown-field error with "renamed to accelerator_pool_id", and pin it in the removed-pool.yaml acceptance case.
| NumAccelerators int `yaml:"num_accelerators" help:"Total number of GPUs to allocate. Must be a positive multiple of the accelerator type's per-node GPU count. See Docs below for GPUs per node." required:"yes" docs:"/machine-learning/ai-runtime/connecting/cli/yaml-config#accelerator-types"` | ||
| AcceleratorType string `yaml:"accelerator_type" help:"Which accelerator to run on, e.g. GPU_1xA10. Matched case-sensitively." required:"yes" docs:"/machine-learning/ai-runtime/connecting/cli/yaml-config#accelerator-types"` | ||
| AcceleratorPoolID *string `yaml:"accelerator_pool_id" help:"GPU Pool to run on, by ID. Must be 1-255 characters. List available pools with 'databricks air pools list'." group:"GPU Pools (optional)"` | ||
| PriorityClass *string `yaml:"priority_class" help:"Scheduling priority within the GPU Pool. Requires accelerator_pool_id." group:"GPU Pools (optional)"` |
There was a problem hiding this comment.
problem: -h config.compute.priority_class no longer lists the valid values (main did), so users only find them by hitting the error.
suggested fix: keep main's list with the new field name: Scheduling priority within the GPU Pool: BEST_EFFORT (lowest, preemptable) or NORMAL. Requires accelerator_pool_id.
There was a problem hiding this comment.
I'd still like a pointer for pool_id. It shipped in v1.18.0 through v1.26.0, so anyone who used it gets unknown field "compute.pool_id" and nothing naming accelerator_pool_id.
Sure. But no one is using it. It's not something folks used before. I would like a clean break rather than referencing the placeholder.
suggested fix: keep main's list with the new field name: Scheduling priority within the GPU Pool: BEST_EFFORT (lowest, preemptable) or NORMAL. Requires accelerator_pool_id.
No, some of those values are deprecated. BEST_EFFORT is already deprecated officially so we should really try not to reference it.
There was a problem hiding this comment.
Priorities are not enabled for users right now
There was a problem hiding this comment.
We can list just NORMAL for now which is implicit anyways which is why I don't love even mentioning it. It's confusing
There was a problem hiding this comment.
But no one is using it. It's not something folks used before. I would like a clean break rather than referencing the placeholder.
aaaah... ok sorry I didn't know that. yes clean break sgtm.
sorry for the frustration caused. my agent proposed it and it looked like a reasonable concern but I didn't have the context that it hasn't been used at all... it was saying it was released before? but I guess it was hidden and / or flag gated somehow?
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
thanks for adding the pool_id hint. listing only NORMAL in the help is fine by me, since BEST_EFFORT is deprecated.
| === unsupported priority is rejected | ||
|
|
||
| >>> [CLI] air run -f invalid-priority.yaml --dry-run | ||
| Error: compute.priority_class: invalid priority_class "HIGH": valid values are: BEST_EFFORT, NORMAL |
There was a problem hiding this comment.
non-blocking: the error still lists BEST_EFFORT, so anyone who mistypes the value (e.g. HIGH here) gets pointed at the deprecated one you dropped from the help.
suggested fix: have validPriorityClassesHint list only NORMAL, and keep accepting BEST_EFFORT so existing YAML still parses.
see #7698 (comment) |
Changes
PRD has naming alignment
Merge blocked pending backend deployment and staging E2E validation.
Rename
compute.pool_idtocompute.accelerator_pool_idfor AIR GPU Pool selection. This is a breaking YAML change:pool_idis rejected with a pointer toaccelerator_pool_id, including when supplied alongside the new selector. Preflight and submission sendaccelerator_pool_id; bundle conversion usesprovisioned_capacity_id, which its current schema supports.Use consistent GPU Pool naming and describe pool selection without promising backend fallback behavior. Priority validation retains the existing supported values.
Why
Align AIR workload configuration with the task-scoped GPU Pool API. See the PRD for the naming decision. Backend support must be deployed before this CLI change merges or releases.
Tests
This PR was written with Codex.