Skip to content

air: find the AI Runtime task anywhere in a multi-task job in air list and air get - #7632

Merged
dimk0 merged 9 commits into
mainfrom
cnxt-2622-air-multitask
Nov 19, 2026
Merged

dimk0 merged 9 commits into
mainfrom
cnxt-2622-air-multitask

Conversation

@dimk0

@dimk0 dimk0 commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Changes

Multi-task AIR jobs are supported, and the official DAB example (universe ai-compute/dabs/examples/03_multitask_dag) runs a notebook prep task before its ai_runtime_task. air list and air get read tasks[0], so air list --all-status listed that run with null experiment/accelerators and air get rendered it with N/A Experiment and Accelerators. (A bundle-deployed job runs as a JOB_RUN, which only --all-status reaches, through the run index; the default air list, --all-users and cancel --all scan SUBMIT_RUN runs only, and this PR doesn't change that.)

Selection rule:

  • Discovery. Walk the run's tasks in task order and pick the first AI Runtime task: an ai_runtime_task, a legacy gen_ai_compute_task with a training script, or a for_each_task whose inner task is one. A run is an AIR run iff it has at least one such task. Non-AIR tasks (the prep notebook) no longer feed status, timing, experiment, accelerators or the MLflow lookup: the run is narrowed to the selected task's task runs (all attempts) before the display helpers read it.
  • Several AIR tasks. air list shows one row per AIR task. Rows share run_id, and the text table names the task in the Run ID column (123 train); JSON gains task_key (null for a run without an AIR task, as in air get). Each row's fields and MLflow link come from its own task runs, including its own status and timing (a skipped task keeps its state and reports no timing), and the experiment/accelerator filters apply per task. A run whose only task is the AIR task keeps its run-level state and timing, so single-task air run output is unchanged. The index-path row cache stores the whole set per run (air-list-runs-v3). cancel --all cancels a multi-row run once.
  • air get. New --task-key flag; default is the first AIR task. A key that is not one of the run's AIR tasks is an INVALID_ARGS error listing them. JSON gains task_key; the text view adds a Task line only when the run has several AIR tasks. The Environment cell reads the selected task's environment_key. The air list detail pane opens the selected row's task.
  • foreach. Unchanged: a for_each_task wrapping an AIR task is that AIR task (sweep view / is_sweep). A job mixing a direct AIR task and a foreach AIR task now yields one row (or one selectable task) each, and air list and air get agree on the default (today list inspected tasks[0] while get picked any foreach).

Out of scope: air logs keeps its last-task behavior; --download-to and --node validation now read the node count from that same task.

Why

Follows the single-task fix in #7587.

Tests

Unit tests for the five requested shapes (AIR task first, AIR task second, two AIR tasks, foreach AIR task, no AIR task) plus retries sharing a key, run narrowing, per-task list rows and filters, air get --task-key selection and error, the detail pane, the legacy gen_ai_compute_task path and the per-task row cache. New acceptance tests acceptance/air/get-multitask (per-task status, environment and --task-key errors) acceptance/air/get-multitask-skipped (a task skipped because prep failed reports UPSTREAM_FAILED and no timing) and acceptance/air/get-run-state (a single-task run keeps its run-level TIMEDOUT state). The air list fixture gains the notebook-first DAB shape, a run with two AI Runtime tasks (one row per task, named in the Run ID column), and an empty -o json list.

Verified against a real run of 03_multitask_dag deployed to e2-dogfood (run 47834518189346) with a build of this branch and a build of main:

main this branch
air list --all-status -o json experiment_name: null, accelerator_type: null air-byot-multitask-dag, GPU_1xA10 ×1, task_key: train
air get Experiment N/A, Accelerators N/A air-byot-multitask-dag, 1x A10
air get --task-key prep n/a run … has no AI Runtime task "prep"; its AI Runtime tasks are: train

go test ./cmd/air, go test ./acceptance -run TestAccept/air and golangci-lint run ./cmd/air/... pass.

This pull request and its description were written by Isaac.

@dimk0
dimk0 requested review from a team as code owners November 14, 2026 04:37
@dimk0
dimk0 requested a review from rclarey November 14, 2026 04:37
dimk0 added a commit that referenced this pull request Nov 14, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Commit: bf8d601b2 | Workflow run

Regression test report

Selected 11 changed tests (limit=20, 0 not selected)

11 tests: regression=11 unreleased=0 coverage=0 failing=0
test branch main (749c68f) latest (v1.26.0)
TestAccept/air/control-chars ✅ ❌ ❌
TestAccept/air/get ✅ ❌ ❌
TestAccept/air/get-ai-runtime ✅ ❌ ❌
TestAccept/air/get-mlflow-retry ✅ ❌ ❌
TestAccept/air/get-multitask ✅ ❌ ❌
TestAccept/air/get-multitask-skipped ✅ ❌ ❌
TestAccept/air/get-pending ✅ ❌ ❌
TestAccept/air/get-run-state ✅ ❌ ❌
TestAccept/air/help ✅ ❌ ❌
TestAccept/air/list ✅ ❌ ❌
TestAccept/air/list-index-fallback ✅ ❌ ❌
TestAccept/air/control-chars: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌

main (749c68f):

acceptance_test.go:1388: Diff:
        --- air/control-chars/output.txt
        +++ /tmp/TestAcceptaircontrol-chars3673986474/002/output.txt
        @@ -38,7 +38,6 @@
             "runs": [
               {
                 "run_id": "123",
        -        "task_key": "train",
                 "run_name": "qwen\u001b[2J-train",
                 "user": "user\u001b[31m@example.com",
                 "status": "SUCCESS",

latest (v1.26.0):

acceptance_test.go:1388: Diff:
        --- air/control-chars/output.txt
        +++ /tmp/TestAcceptaircontrol-chars3790096841/002/output.txt
        @@ -9,13 +9,14 @@
         │  Status        ● SUCCESS                                       │
         │  Submitted     2023-11-14 22:13 UTC                            │
         │  Retries       0                                               │
        -│  Max Retries   N/A                                             │
        +│  Max Retries   0                                               │
         │  Duration      12s                                             │
        -│  Experiment    my-exp                                          │
        -│  MLflow Run    my-run                                          │
        -│  User          user@example.com                                │
        +│  Experiment    my�]0;pwned�-exp                                          │
        +│  MLflow Run    �]0;pwned�my�[2J-run
        +                                          │
        +│  User          user�[31m@example.com                                │
         │  Accelerators  1x A10                                          │
        -│  Environment   databricks_ai_v6                                │
        +│  Environment   N/A                                             │
         │                                                                │
         ╰────────────────────────────────────────────────────────────────╯
         
        @@ -26,7 +25,8 @@
         
         >>> [CLI] air list --all-users
            Run ID  Experiment  Status     Started              Duration  Progress  MLflow                  User              Accelerators
        -   123     my-exp      ● SUCCESS  [TIMESTAMP]       12s         -  my-run                  user@example.com  1x A10
        +   123     my�]0;pwned�-exp      ● SUCCESS  [TIMESTAMP]       12s         -  �]0;pwned�my�[2J-run
        +                  user�[31m@example.com  1x A10
         
         === list (json): strings are passed through as the server sent them
         
        @@ -38,17 +38,11 @@
             "runs": [
               {
                 "run_id": "123",
        -        "task_key": "train",
                 "run_name": "qwen\u001b[2J-train",
                 "user": "user\u001b[31m@example.com",
                 "status": "SUCCESS",
                 "started_at": "[TIMESTAMP]",
        -        "is_sweep": false,
        -        "experiment_name": "my\u001b]0;pwned\u0007-exp",
        -        "duration_seconds": 12,
        -        "accelerator_type": "GPU_1xA10",
        -        "num_accelerators": 1,
        -        "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]"
        +        "is_sweep": false
               }
             ]
           }
TestAccept/air/get: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌

main (749c68f):

acceptance_test.go:1388: Diff:
        --- air/get/output.txt
        +++ /tmp/TestAcceptairget1640278637/002/output.txt
        @@ -43,7 +43,6 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": null,
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,

latest (v1.26.0):

acceptance_test.go:1388: Diff:
        --- air/get/output.txt
        +++ /tmp/TestAcceptairget3555911747/002/output.txt
        @@ -22,7 +22,7 @@
         │  Status        ● SUCCESS                                       │
         │  Submitted     2023-11-14 22:13 UTC                            │
         │  Retries       0                                               │
        -│  Max Retries   N/A                                             │
        +│  Max Retries   0                                               │
         │  Duration      12s                                             │
         │  Experiment    my-exp                                          │
         │  MLflow Run    my-run                                          │
        @@ -43,26 +43,15 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": null,
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
             "attempt_number": 0,
             "experiment_name": "my-exp",
             "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        +    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/artifacts/logs/node_0",
             "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        -    "environment_version": null,
        -    "base_environment": null,
        -    "max_retries": null,
        -    "mlflow_run_name": "my-run",
        -    "sweep": null
        +    "est_percent_complete": null
           }
         }
         
        
FAIL
TestAccept/air/get-ai-runtime: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌

main (749c68f):

acceptance_test.go:1388: Diff:
        --- air/get-ai-runtime/output.txt
        +++ /tmp/TestAcceptairget-ai-runtime3103863290/002/output.txt
        @@ -43,7 +43,6 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
        @@ -74,7 +73,6 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
        @@ -144,7 +142,6 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
        @@ -175,7 +172,6 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
        
    acceptance_test.go:1166: LOG.import: 
    acceptance_test.go:1166: LOG.import-max-retries: 
    acceptance_test.go:1166: LOG.mkdirs: 
FAIL

latest (v1.26.0):

acceptance_test.go:1388: Diff:
        --- air/get-ai-runtime/output.txt
        +++ /tmp/TestAcceptairget-ai-runtime3382212584/002/output.txt
        @@ -22,7 +22,7 @@
         │  Status        ● SUCCESS                                       │
         │  Submitted     2023-11-14 22:13 UTC                            │
         │  Retries       0                                               │
        -│  Max Retries   3 (CLI default)                                 │
        +│  Max Retries   0                                               │
         │  Duration      12s                                             │
         │  Experiment    my-exp                                          │
         │  MLflow Run    my-run                                          │
        @@ -43,162 +43,34 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
             "attempt_number": 0,
             "experiment_name": "my-exp",
             "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        +    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/artifacts/logs/node_0",
             "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        -    "environment_version": "4",
        -    "base_environment": null,
        -    "max_retries": null,
        -    "mlflow_run_name": "my-run",
        -    "sweep": null
        +    "est_percent_complete": null
           }
         }
         
         === get (json, include config)
         
         >>> [CLI] air get 123 -o json --include-config
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "train",
        -    "status": "SUCCESS",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 12,
        -    "attempt_number": 0,
        -    "experiment_name": "my-exp",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        -    "environment_version": "4",
        -    "base_environment": null,
        -    "max_retries": {
        -      "limit": 3,
        -      "source": "cli_default"
        -    },
        -    "mlflow_run_name": "my-run",
        -    "sweep": null,
        -    "config": "experiment_name: my-exp\ncompute:\n  accelerator_type: a10\n  num_accelerators: 1\ncommand: |-\n  for i in $(seq 1 10); do\n    echo \"step $i\"\n  done\n",
        -    "config_source": "uploaded"
        -  }
        -}
        +Error: unknown flag: --include-config
         
        -=== get (text, max_retries set in the config)
        +Usage:
        +  databricks air get JOB_RUN_ID [flags]
         
        ->>> [CLI] air get 123
        +Flags:
        +  -h, --help   help for get
         
        -╭─ Configuration ────────────────────────────────────────────────╮
        -│                                                                │
        -│  experiment_name: my-exp                                       │
        -│  compute:                                                      │
        -│    accelerator_type: a10                                       │
        -│    num_accelerators: 1                                         │
        -│  max_retries: 1                                                │
        -│  command: echo ok                                              │
        -│                                                                │
        -╰────────────────────────────────────────────────────────────────╯
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        -╭─ Metadata ─────────────────────────────────────────────────────╮
        -│                                                                │
        -│  Run ID        123                                             │
        -│  Status        ● SUCCESS                                       │
        -│  Submitted     2023-11-14 22:13 UTC                            │
        -│  Retries       0                                               │
        -│  Max Retries   1                                               │
        -│  Duration      12s                                             │
        -│  Experiment    my-exp                                          │
        -│  MLflow Run    my-run                                          │
        -│  User          user@example.com                                │
        -│  Accelerators  1x A10                                          │
        -│  Environment   4                                               │
        -│                                                                │
        -╰────────────────────────────────────────────────────────────────╯
         
        -Run URL:    [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
        -MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
        -
        -=== get (json, max_retries set in the config)
        -
        ->>> [CLI] air get 123 -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "train",
        -    "status": "SUCCESS",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 12,
        -    "attempt_number": 0,
        -    "experiment_name": "my-exp",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        -    "environment_version": "4",
        -    "base_environment": null,
        -    "max_retries": null,
        -    "mlflow_run_name": "my-run",
        -    "sweep": null
        -  }
        -}
        -
        -=== get (json, include config, max_retries set)
        -
        ->>> [CLI] air get 123 -o json --include-config
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "train",
        -    "status": "SUCCESS",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 12,
        -    "attempt_number": 0,
        -    "experiment_name": "my-exp",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        -    "environment_version": "4",
        -    "base_environment": null,
        -    "max_retries": {
        -      "limit": 1,
        -      "source": "configured"
        -    },
        -    "mlflow_run_name": "my-run",
        -    "sweep": null,
        -    "config": "experiment_name: my-exp\ncompute:\n  accelerator_type: a10\n  num_accelerators: 1\nmax_retries: 1\ncommand: echo ok\n",
        -    "config_source": "uploaded"
        -  }
        -}
        +Exit code: 1
        
    acceptance_test.go:1166: LOG.import: 
    acceptance_test.go:1166: LOG.mkdirs:
TestAccept/air/get-mlflow-retry: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌

main (749c68f):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n=== get of a failed retry with no mlflow output (json)\n\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"task_key\": null,\n    \"status\": \"FAILED\",\n    \"started_at\": \"[TIMESTAMP]\",\n    \"duration_seconds\": 12,\n    \"attempt_number\": 2,\n    \"experiment_name\": null,\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"termination_reason\": \"Out of capacity\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": null,\n    \"accelerator_type\": null,\n    \"num_accelerators\": null,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": null,\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	actual  : "\n=== get of a failed retry with no mlflow output (json)\n\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"FAILED\",\n    \"started_at\": \"[TIMESTAMP]\",\n    \"duration_seconds\": 12,\n    \"attempt_number\": 2,\n    \"experiment_name\": null,\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"termination_reason\": \"Out of capacity\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": null,\n    \"accelerator_type\": null,\n    \"num_accelerators\": null,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": null,\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,3 +9,2 @@
        	            	     "run_id": "123",
        	            	-    "task_key": null,
        	            	     "status": "FAILED",
        	Test:       	TestAccept/air/get-mlflow-retry
        	Messages:   	air/get-mlflow-retry/output.txt vs /tmp/TestAcceptairget-mlflow-retry1306165498/002/output.txt

latest (v1.26.0):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n=== get of a failed retry with no mlflow output (json)\n\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"task_key\": null,\n    \"status\": \"FAILED\",\n    \"started_at\": \"[TIMESTAMP]\",\n    \"duration_seconds\": 12,\n    \"attempt_number\": 2,\n    \"experiment_name\": null,\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"termination_reason\": \"Out of capacity\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": null,\n    \"accelerator_type\": null,\n    \"num_accelerators\": null,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": null,\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	actual  : "\n=== get of a failed retry with no mlflow output (json)\n\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"FAILED\",\n    \"started_at\": \"[TIMESTAMP]\",\n    \"duration_seconds\": 12,\n    \"attempt_number\": 2,\n    \"experiment_name\": null,\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"termination_reason\": \"Out of capacity\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,3 +9,2 @@
        	            	     "run_id": "123",
        	            	-    "task_key": null,
        	            	     "status": "FAILED",
        	            	@@ -17,15 +16,5 @@
        	            	     "mlflow_url": null,
        	            	-    "mlflow_experiment_id": null,
        	            	-    "mlflow_run_id": null,
        	            	     "termination_reason": "Out of capacity",
        	            	     "est_remaining_seconds": null,
        	            	-    "est_percent_complete": null,
        	            	-    "user": null,
        	            	-    "accelerator_type": null,
        	            	-    "num_accelerators": null,
        	            	-    "environment_version": null,
        	            	-    "base_environment": null,
        	            	-    "max_retries": null,
        	            	-    "mlflow_run_name": null,
        	            	-    "sweep": null
        	            	+    "est_percent_complete": null
        	            	   }
        	Test:       	TestAccept/air/get-mlflow-retry
        	Messages:   	air/get-mlflow-retry/output.txt vs /tmp/TestAcceptairget-mlflow-retry2781008436/002/output.txt
TestAccept/air/get-multitask: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌

main (749c68f):

acceptance_test.go:1388: Diff:
        --- air/get-multitask/output.txt
        +++ /tmp/TestAcceptairget-multitask697237340/002/output.txt
        @@ -6,137 +6,41 @@
         ╭─ Metadata ─────────────────────────────────────────────────────╮
         │                                                                │
         │  Run ID        123                                             │
        -│  Task          train                                           │
        -│  Status        ● SUCCESS                                       │
        -│  Submitted     2023-11-14 22:13 UTC                            │
        -│  Retries       0                                               │
        -│  Max Retries   0                                               │
        -│  Duration      10s                                             │
        -│  Experiment    air-byot-multitask-dag                          │
        -│  MLflow Run    matmul-smoke                                    │
        -│  User          user@example.com                                │
        -│  Accelerators  1x A10                                          │
        -│  Environment   5                                               │
        -│                                                                │
        -╰────────────────────────────────────────────────────────────────╯
        -
        -Run URL:    [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
        -MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
        -
        -=== get --task-key eval (text)
        -
        ->>> [CLI] air get 123 --task-key eval
        -
        -╭─ Metadata ─────────────────────────────────────────────────────╮
        -│                                                                │
        -│  Run ID        123                                             │
        -│  Task          eval                                            │
         │  Status        ● FAILED                                        │
         │  Submitted     2023-11-14 22:13 UTC                            │
         │  Retries       0                                               │
         │  Max Retries   2                                               │
         │  Duration      8s                                              │
        -│  Experiment    eval-exp                                        │
        +│  Experiment    N/A                                             │
         │  MLflow Run    matmul-smoke                                    │
         │  User          user@example.com                                │
        -│  Accelerators  8x H100                                         │
        -│  Environment   6                                               │
        +│  Accelerators  N/A                                             │
        +│  Environment   5                                               │
         │                                                                │
         ╰────────────────────────────────────────────────────────────────╯
         
        -Failure reason: eval: loss is NaN (step 120)
        +Failure reason: Task eval failed with message: eval: loss is NaN
         
         Run URL:    [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
         MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
         
        -=== get (json)
        +=== get --task-key eval (text)
         
        ->>> [CLI] air get 123 -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "train",
        -    "status": "SUCCESS",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 10,
        -    "attempt_number": 0,
        -    "experiment_name": "air-byot-multitask-dag",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        -    "environment_version": "5",
        -    "base_environment": null,
        -    "max_retries": {
        -      "limit": 0,
        -      "source": "reported"
        -    },
        -    "mlflow_run_name": "matmul-smoke",
        -    "sweep": null
        -  }
        -}
        +>>> [CLI] air get 123 --task-key eval
        +Error: unknown flag: --task-key
         
        -=== get --task-key eval (json)
        +Usage:
        +  databricks air get JOB_RUN_ID [flags]
         
        ->>> [CLI] air get 123 --task-key eval -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "eval",
        -    "status": "FAILED",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 8,
        -    "attempt_number": 0,
        -    "experiment_name": "eval-exp",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        -    "termination_reason": "eval: loss is NaN (step 120)",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_8xH100",
        -    "num_accelerators": 8,
        -    "environment_version": "6",
        -    "base_environment": null,
        -    "max_retries": {
        -      "limit": 2,
        -      "source": "reported"
        -    },
        -    "mlflow_run_name": "matmul-smoke",
        -    "sweep": null
        -  }
        -}
        +Flags:
        +  -h, --help[=true|false]             help for get
        +      --include-config[=true|false]   Include stored config in JSON (downloads over 1 MiB are omitted; may contain plaintext env_variables and parameters)
         
        -=== get --task-key prep: not an AI Runtime task
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
         
        ->>> [CLI] air get 123 --task-key prep
        -Error: run 123 has no AI Runtime task "prep"; its AI Runtime tasks are: train, eval
         
         Exit code: 1
        -
        -=== get --task-key prep (json)
        -
        ->>> [CLI] air get 123 --task-key prep -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "INVALID_ARGS",
        -    "kind": "PERMANENT",
        -    "message": "run 123 has no AI Runtime task \"prep\"; its AI Runtime tasks are: train, eval",
        -    "retryable": false
        -  }
        -}
        -
        -Exit code: 1

latest (v1.26.0):

acceptance_test.go:1388: Diff:
        --- air/get-multitask/output.txt
        +++ /tmp/TestAcceptairget-multitask1734278785/002/output.txt
        @@ -6,137 +6,40 @@
         ╭─ Metadata ─────────────────────────────────────────────────────╮
         │                                                                │
         │  Run ID        123                                             │
        -│  Task          train                                           │
        -│  Status        ● SUCCESS                                       │
        -│  Submitted     2023-11-14 22:13 UTC                            │
        -│  Retries       0                                               │
        -│  Max Retries   0                                               │
        -│  Duration      10s                                             │
        -│  Experiment    air-byot-multitask-dag                          │
        -│  MLflow Run    matmul-smoke                                    │
        -│  User          user@example.com                                │
        -│  Accelerators  1x A10                                          │
        -│  Environment   5                                               │
        -│                                                                │
        -╰────────────────────────────────────────────────────────────────╯
        -
        -Run URL:    [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
        -MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
        -
        -=== get --task-key eval (text)
        -
        ->>> [CLI] air get 123 --task-key eval
        -
        -╭─ Metadata ─────────────────────────────────────────────────────╮
        -│                                                                │
        -│  Run ID        123                                             │
        -│  Task          eval                                            │
         │  Status        ● FAILED                                        │
         │  Submitted     2023-11-14 22:13 UTC                            │
         │  Retries       0                                               │
         │  Max Retries   2                                               │
         │  Duration      8s                                              │
        -│  Experiment    eval-exp                                        │
        +│  Experiment    N/A                                             │
         │  MLflow Run    matmul-smoke                                    │
         │  User          user@example.com                                │
        -│  Accelerators  8x H100                                         │
        -│  Environment   6                                               │
        +│  Accelerators  N/A                                             │
        +│  Environment   5                                               │
         │                                                                │
         ╰────────────────────────────────────────────────────────────────╯
         
        -Failure reason: eval: loss is NaN (step 120)
        +Failure reason: Task eval failed with message: eval: loss is NaN
         
         Run URL:    [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
         MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
         
        -=== get (json)
        +=== get --task-key eval (text)
         
        ->>> [CLI] air get 123 -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "train",
        -    "status": "SUCCESS",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 10,
        -    "attempt_number": 0,
        -    "experiment_name": "air-byot-multitask-dag",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        -    "environment_version": "5",
        -    "base_environment": null,
        -    "max_retries": {
        -      "limit": 0,
        -      "source": "reported"
        -    },
        -    "mlflow_run_name": "matmul-smoke",
        -    "sweep": null
        -  }
        -}
        +>>> [CLI] air get 123 --task-key eval
        +Error: unknown flag: --task-key
         
        -=== get --task-key eval (json)
        +Usage:
        +  databricks air get JOB_RUN_ID [flags]
         
        ->>> [CLI] air get 123 --task-key eval -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "eval",
        -    "status": "FAILED",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 8,
        -    "attempt_number": 0,
        -    "experiment_name": "eval-exp",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "mlflow_experiment_id": "exp1",
        -    "mlflow_run_id": "run1",
        -    "termination_reason": "eval: loss is NaN (step 120)",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_8xH100",
        -    "num_accelerators": 8,
        -    "environment_version": "6",
        -    "base_environment": null,
        -    "max_retries": {
        -      "limit": 2,
        -      "source": "reported"
        -    },
        -    "mlflow_run_name": "matmul-smoke",
        -    "sweep": null
        -  }
        -}
        +Flags:
        +  -h, --help   help for get
         
        -=== get --task-key prep: not an AI Runtime task
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        ->>> [CLI] air get 123 --task-key prep
        -Error: run 123 has no AI Runtime task "prep"; its AI Runtime tasks are: train, eval
         
         Exit code: 1
        -
        -=== get --task-key prep (json)
        -
        ->>> [CLI] air get 123 --task-key prep -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "INVALID_ARGS",
        -    "kind": "PERMANENT",
        -    "message": "run 123 has no AI Runtime task \"prep\"; its AI Runtime tasks are: train, eval",
        -    "retryable": false
        -  }
        -}
        -
        -Exit code: 1
TestAccept/air/get-multitask-skipped: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌

main (749c68f):

acceptance_test.go:1388: Diff:
        --- air/get-multitask-skipped/output.txt
        +++ /tmp/TestAcceptairget-multitask-skipped46796686/002/output.txt
        @@ -6,19 +6,21 @@
         ╭─ Metadata ─────────────────────────────────────────────────────╮
         │                                                                │
         │  Run ID        123                                             │
        -│  Status        ● UPSTREAM_FAILED                               │
        -│  Submitted     N/A                                             │
        +│  Status        ● FAILED                                        │
        +│  Submitted     2023-11-14 22:13 UTC                            │
         │  Retries       0                                               │
         │  Max Retries   0                                               │
        -│  Duration      N/A                                             │
        -│  Experiment    air-byot-multitask-dag                          │
        +│  Duration      5s                                              │
        +│  Experiment    N/A                                             │
         │  MLflow Run    N/A                                             │
         │  User          user@example.com                                │
        -│  Accelerators  1x A10                                          │
        +│  Accelerators  N/A                                             │
         │  Environment   N/A                                             │
         │                                                                │
         ╰────────────────────────────────────────────────────────────────╯
         
        +Failure reason: Task prep failed
        +
         Run URL:    [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
         
         === get (json)
        @@ -29,21 +23,21 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
        -    "status": "UPSTREAM_FAILED",
        -    "started_at": null,
        -    "duration_seconds": null,
        +    "status": "FAILED",
        +    "started_at": "[TIMESTAMP]",
        +    "duration_seconds": 5,
             "attempt_number": 0,
        -    "experiment_name": "air-byot-multitask-dag",
        +    "experiment_name": null,
             "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
             "mlflow_url": null,
             "mlflow_experiment_id": null,
             "mlflow_run_id": null,
        +    "termination_reason": "Task prep failed",
             "est_remaining_seconds": null,
             "est_percent_complete": null,
             "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        +    "accelerator_type": null,
        +    "num_accelerators": null,
             "environment_version": null,
             "base_environment": null,
             "max_retries": {

latest (v1.26.0):

acceptance_test.go:1388: Diff:
        --- air/get-multitask-skipped/output.txt
        +++ /tmp/TestAcceptairget-multitask-skipped3008347824/002/output.txt
        @@ -6,19 +6,21 @@
         ╭─ Metadata ─────────────────────────────────────────────────────╮
         │                                                                │
         │  Run ID        123                                             │
        -│  Status        ● UPSTREAM_FAILED                               │
        -│  Submitted     N/A                                             │
        +│  Status        ● FAILED                                        │
        +│  Submitted     2023-11-14 22:13 UTC                            │
         │  Retries       0                                               │
         │  Max Retries   0                                               │
        -│  Duration      N/A                                             │
        -│  Experiment    air-byot-multitask-dag                          │
        +│  Duration      5s                                              │
        +│  Experiment    N/A                                             │
         │  MLflow Run    N/A                                             │
         │  User          user@example.com                                │
        -│  Accelerators  1x A10                                          │
        +│  Accelerators  N/A                                             │
         │  Environment   N/A                                             │
         │                                                                │
         ╰────────────────────────────────────────────────────────────────╯
         
        +Failure reason: Task prep failed
        +
         Run URL:    [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
         
         === get (json)
        @@ -29,28 +23,15 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
        -    "status": "UPSTREAM_FAILED",
        -    "started_at": null,
        -    "duration_seconds": null,
        +    "status": "FAILED",
        +    "started_at": "[TIMESTAMP]",
        +    "duration_seconds": 5,
             "attempt_number": 0,
        -    "experiment_name": "air-byot-multitask-dag",
        +    "experiment_name": null,
             "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
             "mlflow_url": null,
        -    "mlflow_experiment_id": null,
        -    "mlflow_run_id": null,
        +    "termination_reason": "Task prep failed",
             "est_remaining_seconds": null,
        -    "est_percent_complete": null,
        -    "user": "user@example.com",
        -    "accelerator_type": "GPU_1xA10",
        -    "num_accelerators": 1,
        -    "environment_version": null,
        -    "base_environment": null,
        -    "max_retries": {
        -      "limit": 0,
        -      "source": "reported"
        -    },
        -    "mlflow_run_name": null,
        -    "sweep": null
        +    "est_percent_complete": null
           }
         }
TestAccept/air/get-pending: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌

main (749c68f):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"task_key\": \"train\",\n    \"status\": \"PENDING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"status_message\": \"Waiting for accelerator compute capacity to become available...\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": \"user@example.com\",\n    \"accelerator_type\": \"GPU_1xA10\",\n    \"num_accelerators\": 1,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": {\n      \"limit\": 0,\n      \"source\": \"reported\"\n    },\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	actual  : "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"PENDING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"status_message\": \"Waiting for accelerator compute capacity to become available...\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": \"user@example.com\",\n    \"accelerator_type\": \"GPU_1xA10\",\n    \"num_accelerators\": 1,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": {\n      \"limit\": 0,\n      \"source\": \"reported\"\n    },\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,3 +7,2 @@
        	            	     "run_id": "123",
        	            	-    "task_key": "train",
        	            	     "status": "PENDING",
        	Test:       	TestAccept/air/get-pending
        	Messages:   	air/get-pending/output.txt vs /tmp/TestAcceptairget-pending1156729743/002/output.txt

latest (v1.26.0):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"task_key\": \"train\",\n    \"status\": \"PENDING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"status_message\": \"Waiting for accelerator compute capacity to become available...\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": \"user@example.com\",\n    \"accelerator_type\": \"GPU_1xA10\",\n    \"num_accelerators\": 1,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": {\n      \"limit\": 0,\n      \"source\": \"reported\"\n    },\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	actual  : "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"RUNNING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,4 +7,3 @@
        	            	     "run_id": "123",
        	            	-    "task_key": "train",
        	            	-    "status": "PENDING",
        	            	+    "status": "RUNNING",
        	            	     "started_at": null,
        	            	@@ -15,18 +14,4 @@
        	            	     "mlflow_url": null,
        	            	-    "mlflow_experiment_id": null,
        	            	-    "mlflow_run_id": null,
        	            	-    "status_message": "Waiting for accelerator compute capacity to become available...",
        	            	     "est_remaining_seconds": null,
        	            	-    "est_percent_complete": null,
        	            	-    "user": "user@example.com",
        	            	-    "accelerator_type": "GPU_1xA10",
        	            	-    "num_accelerators": 1,
        	            	-    "environment_version": null,
        	            	-    "base_environment": null,
        	            	-    "max_retries": {
        	            	-      "limit": 0,
        	            	-      "source": "reported"
        	            	-    },
        	            	-    "mlflow_run_name": null,
        	            	-    "sweep": null
        	            	+    "est_percent_complete": null
        	            	   }
        	Test:       	TestAccept/air/get-pending
        	Messages:   	air/get-pending/output.txt vs /tmp/TestAcceptairget-pending1161595905/002/output.txt
TestAccept/air/get-run-state: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌

main (749c68f):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n=== get (json): the run's TIMEDOUT state and message, not its task's SUCCESS\n\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"task_key\": \"train\",\n    \"status\": \"TIMEDOUT\",\n    \"started_at\": \"[TIMESTAMP]\",\n    \"duration_seconds\": 3500,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"termination_reason\": \"Task train timed out after 3600 seconds\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": \"user@example.com\",\n    \"accelerator_type\": \"GPU_1xA10\",\n    \"num_accelerators\": 1,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": {\n      \"limit\": 0,\n      \"source\": \"reported\"\n    },\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	actual  : "\n=== get (json): the run's TIMEDOUT state and message, not its task's SUCCESS\n\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"TIMEDOUT\",\n    \"started_at\": \"[TIMESTAMP]\",\n    \"duration_seconds\": 3500,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"termination_reason\": \"Task train timed out after 3600 seconds\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": \"user@example.com\",\n    \"accelerator_type\": \"GPU_1xA10\",\n    \"num_accelerators\": 1,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": {\n      \"limit\": 0,\n      \"source\": \"reported\"\n    },\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,3 +9,2 @@
        	            	     "run_id": "123",
        	            	-    "task_key": "train",
        	            	     "status": "TIMEDOUT",
        	Test:       	TestAccept/air/get-run-state
        	Messages:   	air/get-run-state/output.txt vs /tmp/TestAcceptairget-run-state2066013315/002/output.txt

latest (v1.26.0):

acceptance_test.go:1388: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1388
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1129
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:674
        	Error:      	Not equal: 
        	            	expected: "\n=== get (json): the run's TIMEDOUT state and message, not its task's SUCCESS\n\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"task_key\": \"train\",\n    \"status\": \"TIMEDOUT\",\n    \"started_at\": \"[TIMESTAMP]\",\n    \"duration_seconds\": 3500,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"mlflow_experiment_id\": null,\n    \"mlflow_run_id\": null,\n    \"termination_reason\": \"Task train timed out after 3600 seconds\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null,\n    \"user\": \"user@example.com\",\n    \"accelerator_type\": \"GPU_1xA10\",\n    \"num_accelerators\": 1,\n    \"environment_version\": null,\n    \"base_environment\": null,\n    \"max_retries\": {\n      \"limit\": 0,\n      \"source\": \"reported\"\n    },\n    \"mlflow_run_name\": null,\n    \"sweep\": null\n  }\n}\n"
        	            	actual  : "\n=== get (json): the run's TIMEDOUT state and message, not its task's SUCCESS\n\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"TIMEDOUT\",\n    \"started_at\": \"[TIMESTAMP]\",\n    \"duration_seconds\": 3500,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"termination_reason\": \"Task train timed out after 3600 seconds\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,3 +9,2 @@
        	            	     "run_id": "123",
        	            	-    "task_key": "train",
        	            	     "status": "TIMEDOUT",
        	            	@@ -17,18 +16,5 @@
        	            	     "mlflow_url": null,
        	            	-    "mlflow_experiment_id": null,
        	            	-    "mlflow_run_id": null,
        	            	     "termination_reason": "Task train timed out after 3600 seconds",
        	            	     "est_remaining_seconds": null,
        	            	-    "est_percent_complete": null,
        	            	-    "user": "user@example.com",
        	            	-    "accelerator_type": "GPU_1xA10",
        	            	-    "num_accelerators": 1,
        	            	-    "environment_version": null,
        	            	-    "base_environment": null,
        	            	-    "max_retries": {
        	            	-      "limit": 0,
        	            	-      "source": "reported"
        	            	-    },
        	            	-    "mlflow_run_name": null,
        	            	-    "sweep": null
        	            	+    "est_percent_complete": null
        	            	   }
        	Test:       	TestAccept/air/get-run-state
        	Messages:   	air/get-run-state/output.txt vs /tmp/TestAcceptairget-run-state795315468/002/output.txt

3 more failure details omitted, see the full report in the run.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: bf8d601

Run: 44394410224

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 30 6:26
💚​ aws-windows-latest-4core-16gb 1 274 28 6:30
💚​ azure linux-2core-8gb 1 271 30 6:26
💚​ azure-windows-latest-4core-16gb 1 273 28 5:15
💚​ gcp linux-2core-8gb 1 272 30 5:42
💚​ gcp-windows-latest-4core-16gb 1 274 28 6:06
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
3:20 gcp-windows-latest-4core-16gb TestAccept
2:48 azure-windows-latest-4core-16gb TestAccept
2:40 aws-windows-latest-4core-16gb TestAccept
2:05 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir

dimk0 added a commit that referenced this pull request Nov 17, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 force-pushed the cnxt-2622-air-multitask branch from 52dfc19 to 06ba7d8 Compare November 17, 2026 03:41
dimk0 added a commit that referenced this pull request Nov 17, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 force-pushed the cnxt-2622-air-multitask branch from 06ba7d8 to efdde9c Compare November 17, 2026 05:41
dimk0 added a commit that referenced this pull request Nov 17, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 force-pushed the cnxt-2622-air-multitask branch from efdde9c to f7bb7ad Compare November 17, 2026 20:17
Comment thread cmd/air/list_format.go Resolved
Comment thread cmd/air/list.go Resolved
Comment thread cmd/air/list_tui.go Resolved
Comment thread cmd/air/get.go Resolved
Comment thread cmd/air/cancel.go Resolved
Comment thread cmd/air/cancel.go Resolved
Comment thread cmd/air/joblist.go Resolved
dimk0 added a commit that referenced this pull request Nov 17, 2026
Address review on #7632:

- A run narrowed to one AI Runtime task now reports that task's latest
  attempt's state, so a succeeded train next to a failed eval, or an AI
  Runtime task skipped after a failed prep notebook, no longer shows the
  run's aggregate. While the run is still active after a failed attempt,
  the run's state stands, since the task may be waiting to retry.
- `--limit` counts runs on both the scan and index paths.
- `task_key` is null for a run without an AI Runtime task, matching
  `air list`.
- The `air list` logs pane notes when its logs belong to another task than
  the selected row's.
- `cancel --all` previews every AI Runtime task of each run, and its help
  says a cancel stops the whole run.
- Tests for per-task status with differing results, the run-counted
  limit, `cancel --all` on a multi-task run, and the logs note.

Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 requested a review from maggiewang-db November 17, 2026 22:39

@pardis-beikzadeh-db pardis-beikzadeh-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for turning Maggie's comments around so quickly. The status, --limit and task_key fixes look right. Before merge, I'd like to see the task lookups, the key check and the per-run grouping each written once. Details inline. Agreed that per-task air logs can be a follow-up.

Comment thread cmd/air/get.go Resolved
Comment thread cmd/air/joblist.go Resolved
Comment thread cmd/air/list.go Resolved
Comment thread cmd/air/get.go Resolved
dimk0 added a commit that referenced this pull request Nov 18, 2026
Address review on #7632:

- `airTaskKeys` is the one scan for AI Runtime tasks; `isAirRun` is built
  on it, and readers take `Tasks[0]` of the run narrowed by `taskRun`
  (`leadTask`), so `airTask` and `firstAirTaskIndex` are gone. `isAirTask`
  unwraps a foreach once (`taskSpec`) instead of checking twice. Callers
  outside `get` and `list` that read an unnarrowed run go through
  `defaultTaskRun`.
- `selectAirTask` checks the key once with early returns; `taskRun` no
  longer re-checks it. The "no AI Runtime tasks" error no longer repeats.
- `air get` and the detail pane share `taskView`.
- Every list path passes runs as groups (`runGroup`): `--limit` counts
  groups, the index path sorts groups so a run's rows stay together,
  filtering happens in `matchingGroup`, the cache stores and reads a
  run's entries in one place, and `cancel --all` takes each group's first
  row and task keys. `countRuns` and `firstRowPerRun` are gone.

Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0

dimk0 commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

@pardis-beikzadeh-db thanks. a081563 writes each of these once:

  • Task lookups: airTaskKeys is the one scan, and readers take the narrowed run's Tasks[0].
  • Key check: only in selectAirTask.
  • Per-run grouping: runGroup, used by --limit, sorting, filtering, the cache and cancel --all.

Details are in each thread. Per-task air logs stays a follow-up, as agreed.

dimk0 and others added 4 commits November 18, 2026 01:01
…st` and `air get`

Multi-task AIR jobs, such as the official DAB example that runs a notebook
prep task before its ai_runtime_task, were omitted from `air list` and
showed N/A experiment and accelerators in `air get`, because both read
tasks[0].

- Discover the first AIR task in task order: an ai_runtime_task, a legacy
  gen_ai_compute_task with a training script, or a foreach wrapping one.
  Non-AIR tasks no longer feed status, timing or the MLflow lookup.
- A job with several AIR tasks lists one row per task. Rows share the
  run_id; JSON output gains task_key. Each row reads its own task runs.
- `air get` gains --task-key (default: the first AIR task), its JSON gains
  task_key, and the text view adds a Task line when the run has several.
- The list detail pane opens the selected row's task; `cancel --all`
  cancels a multi-row run once.
- The index-path row cache stores one row per AIR task (air-list-runs-v3).

Verified against a real run of universe ai-compute/dabs/examples/03_multitask_dag
on e2-dogfood (run 47834518189346).

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Address review on #7632:

- A run narrowed to one AI Runtime task now reports that task's latest
  attempt's state, so a succeeded train next to a failed eval, or an AI
  Runtime task skipped after a failed prep notebook, no longer shows the
  run's aggregate. While the run is still active after a failed attempt,
  the run's state stands, since the task may be waiting to retry.
- `--limit` counts runs on both the scan and index paths.
- `task_key` is null for a run without an AI Runtime task, matching
  `air list`.
- The `air list` logs pane notes when its logs belong to another task than
  the selected row's.
- `cancel --all` previews every AI Runtime task of each run, and its help
  says a cancel stops the whole run.
- Tests for per-task status with differing results, the run-counted
  limit, `cancel --all` on a multi-task run, and the logs note.

Co-authored-by: Isaac <no-reply@databricks.com>
Address review on #7632:

- `airTaskKeys` is the one scan for AI Runtime tasks; `isAirRun` is built
  on it, and readers take `Tasks[0]` of the run narrowed by `taskRun`
  (`leadTask`), so `airTask` and `firstAirTaskIndex` are gone. `isAirTask`
  unwraps a foreach once (`taskSpec`) instead of checking twice. Callers
  outside `get` and `list` that read an unnarrowed run go through
  `defaultTaskRun`.
- `selectAirTask` checks the key once with early returns; `taskRun` no
  longer re-checks it. The "no AI Runtime tasks" error no longer repeats.
- `air get` and the detail pane share `taskView`.
- Every list path passes runs as groups (`runGroup`): `--limit` counts
  groups, the index path sorts groups so a run's rows stay together,
  filtering happens in `matchingGroup`, the cache stores and reads a
  run's entries in one place, and `cancel --all` takes each group's first
  row and task keys. `countRuns` and `firstRowPerRun` are gone.

Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 force-pushed the cnxt-2622-air-multitask branch from a081563 to 2e0e1d5 Compare November 18, 2026 01:08
Comment thread cmd/air/list.go Resolved
Comment thread cmd/air/joblist.go Resolved
Comment thread cmd/air/format.go Resolved
Comment thread acceptance/air/list/test.toml Resolved
Comment thread cmd/air/joblist.go Resolved
Comment thread cmd/air/list_format.go Resolved
Comment thread cmd/air/get.go Resolved
Comment thread cmd/air/logdownload.go Resolved
Comment thread cmd/air/list.go Resolved

@pardis-beikzadeh-db pardis-beikzadeh-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the lookups and grouping read much better now. Two things before merge:

  1. +1 to @rclarey's threads, especially "runs": null on an empty list and the state swap on single-task runs. Those change what air run users see today.

  2. The acceptance outputs don't show either change, because no fixture has them. Could we add acceptance cases for:

    • air list -o json with no matching runs
    • air get on a single-task run whose run state differs from its task's (e.g. run TIMEDOUT, task SUCCESS)
    • air list text output for a run with two AI Runtime tasks

    Related: most of the new Go tests call helpers directly (taskRun, selectAirTask, cancelPreviewRows, ...), which is why the last commit had to rewrite eight of them after helpers were renamed. Once the acceptance cases cover the behavior, those can go.

Two small nits inline.

Comment thread cmd/air/get.go Resolved
Comment thread cmd/air/joblist.go Resolved
…vironment, timing and nodes

Review follow-ups for multi-task AI Runtime jobs:

- A run whose only task is the AI Runtime task keeps its run-level state,
  status and timing, so single-task `air run` submissions report as before
  (termination prefix, run-level TIMEDOUT).
- A skipped task (upstream failed, upstream cancelled, excluded) keeps its own
  state, and a task of a multi-task run reports only its own timing, never the
  whole run's.
- Only a pending run, or a running run whose task waits to start, reports as
  waiting for compute; queued, blocked and waiting-for-retry runs report that
  state, as on main.
- `air get` reads the environment keyed by the selected task's
  environment_key instead of `default`.
- `air logs --download-to` and `--node` validation read the node count from
  the task the logs come from.
- `air list` names the task in the Run ID column for a run with several AI
  Runtime tasks, reports `task_key` as null without one (matching `air get`),
  prints `"runs": []` for an empty list, and sorts a run by its first started
  task.
- The list fixture comment no longer implies the default scan reaches
  bundle-deployed (JOB_RUN) jobs.

Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 requested a review from rclarey November 18, 2026 18:20
- acceptance/air/get-run-state: a single-task run that TIMEDOUT after its task
  succeeded reports the run's TIMEDOUT state and termination message.
- acceptance/air/list: a run with two AI Runtime tasks lists as two rows that
  name their task and report their own status, and `air list -o json` with no
  matching runs prints "runs": [].
- Drop the unit tests these and the existing get-multitask and cancel cases now
  cover, which called helpers directly.
- selectAirTask starts from defaultTaskRun for an empty key, and taskRun drops
  a guard its callers make unnecessary.

Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0

dimk0 commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

@pardis-beikzadeh-db, thanks. Both points are addressed:

  1. All of @rclarey's threads are fixed in 0b6c20b and resolved with replies. "runs": null on an empty list is back to []. A run whose only task is the AI Runtime task keeps its run-level state, status and timing, so air run output is unchanged.

  2. The acceptance cases are in ddfaf6c:

    • acceptance/air/list: air list -o json with no matching runs now prints "runs": [].
    • New acceptance/air/get-run-state: air get on a single-task run that TIMEDOUT after its task succeeded reports TIMEDOUT and the run's "Task train timed out…" message.
    • acceptance/air/list: a run with two AI Runtime tasks that ended differently lists as … train (SUCCESS) and … eval (FAILED), in both text and JSON.

    In 0b6c20b, get-multitask also gave eval its own environment, so the golden checks the per-task Environment cell.

On the helper-level tests: I dropped the ones these cases and the existing get-multitask and cancel --all coverage now make redundant. Those are TestTaskViewTaskKey, TestCancelPreviewRows, and the row-render, empty-JSON, null-task_key and single-task-status tests I'd added in the previous push.

I kept the tests for behavior that would need many more fixtures to cover end to end:

  • the state matrix in TestTaskRunReportsTheTasksOwnState (queued, blocked, skipped while another branch runs, retries) and TestRunLevelWaitingStatesAreReportedAsSuch
  • skipped-task timing
  • the --download-to node count
  • the environment-key lookup
  • the per-task filter
  • selectAirTask's error messages, including the no-AI-Runtime-task case, which get-multitask can't reach because its run has AI Runtime tasks

If you'd rather have any of these as acceptance cases instead, I can convert them.

@pardis-beikzadeh-db pardis-beikzadeh-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @rclarey's threads, my two nits and the three acceptance cases all look done. A few non-blocking follow-ups inline.

Comment thread cmd/air/logdownload.go Resolved
Comment thread cmd/air/render.go Resolved
Comment thread cmd/air/joblist_test.go Resolved
Comment thread cmd/air/joblist.go Resolved
…rough firstTask

Follow-ups from review:

- downloadLogs resolves the retry once and passes that task run to
  resolveNodeCount; logTaskRun becomes a plain lookup by task run ID, and
  runLogs's --node check goes through nodeCountForRetry.
- firstTask's synthetic foreach task carries EnvironmentKey, so
  taskEnvironmentKey reads firstTask(run).EnvironmentKey instead of unwrapping
  the sweep a third time.
- acceptance/air/get-multitask-skipped: prep fails and train is skipped, so the
  golden shows UPSTREAM_FAILED and no start time or duration.
- The run-level waiting-state test asserts the exact status.

Co-authored-by: Isaac <no-reply@databricks.com>
Comment thread cmd/air/joblist.go Resolved
Comment thread cmd/air/joblist_test.go Resolved
Comment thread cmd/air/format.go Resolved
Comment thread .nextchanges/air/multitask-air-tasks.md Resolved
…eued task as pending

Review follow-ups:

- The index path keeps a batch in the index's order (newest submission
  first) instead of re-sorting by row start time, which a task that hasn't
  started or was skipped no longer has. Such a run now lists where main lists
  it, fresh or cached.
- A task of a multi-task run that is pending, queued or waiting to retry
  reports PENDING with the waiting-for-compute message, as a single-task run
  does; a task blocked on an upstream task still reports BLOCKED.
- The per-task state cases that had a single task key get a sibling task, so
  they reach taskState, and get-multitask carries status.termination_details
  at run and task level, so the task's own failure reason is pinned.
- The changelog and `air get` help state that a task whose latest attempt
  failed reads RUNNING while another task of the run is still active.

Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 requested a review from rclarey November 18, 2026 20:01
Comment thread cmd/air/joblist.go Resolved
Comment thread cmd/air/list_index.go Resolved
Review follow-ups:

- Only an attempt Jobs may retry (failed, timed out, internal error) makes a
  task of an active run read the run's RUNNING; a cancelled task, like a
  skipped one, keeps its own state.
- Tests pin the index path's order: a newer active run lists above cached
  older ones, and a retried run lists by submission time. The changelog notes
  that `air list --all-status` now orders runs by submission time.

Co-authored-by: Isaac <no-reply@databricks.com>
@dimk0
dimk0 requested a review from rclarey November 18, 2026 20:45
@dimk0
dimk0 added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 562948c Nov 19, 2026
28 checks passed
@dimk0
dimk0 deleted the cnxt-2622-air-multitask branch November 19, 2026 17:30
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: 562948c

Run: 44578154390

Env 💥​PANIC ❌​FAIL 🟨​KNOWN 🤯​MISS 💚​RECOVERED ✅​pass 🙈​skip Time
💥​ aws linux-2core-8gb 1 2 278 18 1331 1416 320:00
💥​ aws-windows-latest-4core-16gb 1 2 219 23 1301 1444 320:00
🟨​ azure linux-2core-8gb 8 231 15 1356 1424 320:00
💥​ azure-windows-latest-4core-16gb 1 2 12 150 14 1347 1452 320:00
💥​ gcp linux-2core-8gb 1 294 20 1302 1422 320:00
💥​ gcp-windows-latest-4core-16gb 1 258 18 1256 1450 319:59
522 interesting tests: 490 MISS, 14 KNOWN, 9 RECOVERED, 5 PANIC, 4 FAIL
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🤯​ TestAccept 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/apps/job_permissions 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/apps/job_permissions/DMS= 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/apps/job_permissions/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist/DMS= 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed/DMS= 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/config_edits 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/config_edits/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/config_edits/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_fields 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_fields/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_fields/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/multiple_files 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/multiple_files/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/multiple_files/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/output_json 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/output_json/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/output_json/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables ✅​p 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables/DMS= ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables/DMS=true ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/select_basic 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/select_basic/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/select_basic/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/select_multiple 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/select_multiple/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/select_multiple/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/target_override 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/target_override/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/target_override/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/debug/fetch-repository-info 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/debug/fetch-repository-info/DMS= 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/debug/fetch-repository-info/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=/DMS= 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=true/DMS= 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=true/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync/DMS=true 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/basic 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/basic/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/breakglass 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/breakglass/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/can-manage 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/can-manage/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/destroy-redeploy 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/destroy-redeploy/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/dev-mode 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/dev-mode/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/legacy-state 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/legacy-state/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/missing-snapshot 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/missing-snapshot/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts/DMS=/READPLAN= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts/DMS=/READPLAN=1 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/permissions-change 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/permissions-change/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/select 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/select/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/mlops-stacks 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/mlops-stacks/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/mlops-stacks/DMS=true 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/spark-jar-task 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/spark-jar-task/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deploy/spark-jar-task/DMS=true 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🟨​ TestAccept/bundle/deployment/bind/alert 🤯​M 💚​R 🤯​M 🟨​K 🤯​M 🤯​M
🟨​ TestAccept/bundle/deployment/bind/alert/DMS= 🤯​M 💚​R 🤯​M 🟨​K 🤯​M 🤯​M
🟨​ TestAccept/bundle/deployment/bind/alert/DMS=true 🤯​M 💚​R 🤯​M 🟨​K 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/catalog 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/catalog/DMS= 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/catalog/DMS=true 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/cluster 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/cluster/DMS= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/cluster/DMS=true 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=/READPLAN= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=/READPLAN=1 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=true/READPLAN= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=true/READPLAN=1 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/recreation 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN= 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/recreation/DMS=/READPLAN=1 🤯​M 🤯​M 🤯​M ✅​p 🤯​M 🤯​M
Top 50 slowest tests (at least 2 minutes):
duration env testname
14:44 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
11:44 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
11:37 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
11:34 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
11:20 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
11:01 gcp linux-2core-8gb TestAccept/bundle/resources/materialization_sets/immutable/DMS=true
10:46 aws-windows-latest-4core-16gb TestAccept/bundle/resources/registered_models/basic/DMS=
10:44 aws-windows-latest-4core-16gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=yes/DMS=/NBOOK=yes/PY=yes/READPLAN=
10:26 azure-windows-latest-4core-16gb TestAccept/bundle/resources/schemas/rename_then_change/DMS=true
10:18 aws linux-2core-8gb TestAccept/bundle/templates/default-python/integration_classic/DMS=/UV_PYTHON=3.10
10:16 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:58 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:55 aws-windows-latest-4core-16gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=no/DMS=/NBOOK=yes/PY=no/READPLAN=1
9:48 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:43 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
9:36 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:31 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:31 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:17 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:13 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:05 aws-windows-latest-4core-16gb TestAccept/bundle/templates/default-python/combinations/serverless/DLT=no/DMS=/NBOOK=yes/PY=yes/READPLAN=1
8:32 azure-windows-latest-4core-16gb TestAccept/bundle/resources/experiments/basic/DMS=true
8:20 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
8:09 gcp linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=yes/DMS=/NBOOK=yes/PY=no/READPLAN=1
8:06 aws-windows-latest-4core-16gb TestAccept/bundle/templates/default-python/combinations/serverless/DLT=no/DMS=/NBOOK=yes/PY=no/READPLAN=
8:06 aws-windows-latest-4core-16gb TestAccept/bundle/templates/default-python/integration_classic/DMS=/UV_PYTHON=3.12
8:01 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
7:58 gcp linux-2core-8gb TestAccept/bundle/resources/materialization_sets/batch/DMS=true
7:57 aws linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=yes/DMS=/NBOOK=no/PY=no/READPLAN=1
7:56 azure-windows-latest-4core-16gb TestAccept/bundle/resources/registered_models/basic/DMS=true
7:55 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:52 aws-windows-latest-4core-16gb TestAccept/bundle/templates/default-python/integration_classic/DMS=/UV_PYTHON=3.11
7:49 aws linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/serverless/DLT=no/DMS=/NBOOK=yes/PY=no/READPLAN=1
7:43 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:40 gcp linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=yes/DMS=/NBOOK=no/PY=no/READPLAN=
7:36 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:31 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:28 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:26 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
7:25 aws linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/serverless/DLT=yes/DMS=/NBOOK=yes/PY=yes/READPLAN=1
7:25 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_endpoints/update_autoscaling/DMS=true
7:25 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:24 aws-windows-latest-4core-16gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=no/DMS=/NBOOK=no/PY=yes/READPLAN=1
7:22 aws linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/serverless/DLT=yes/DMS=/NBOOK=no/PY=yes/READPLAN=
7:15 azure linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=yes/DMS=/NBOOK=yes/PY=no/READPLAN=1
7:10 gcp linux-2core-8gb TestAccept/bundle/resources/schemas/rename_then_change/DMS=true
7:09 gcp linux-2core-8gb TestAccept/bundle/resources/materialization_sets/batch/DMS=
7:08 aws linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=yes/DMS=/NBOOK=no/PY=no/READPLAN=
7:05 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:05 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
(373 table rows omitted to keep the report under 60000 bytes)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants