Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/air/multitask-air-tasks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* `air list` and `air get` now read the AI Runtime task wherever it sits in a multi-task job (for example after a notebook prep task) instead of the first task. A job with several AI Runtime tasks lists one row per task, named in the table's Run ID column (so `run_id` is no longer unique in `air list -o json`, and `--limit` still counts runs); each such row and `air get --task-key` report that task's own status and timing (except that a task whose latest attempt failed shows the run's RUNNING, with no failure reason, while another task of the run is still active, since it may yet be retried), and `air get` shows the selected task's environment. `air get` gains `--task-key`, and both commands report `task_key` in JSON output (`null` for a run without an AI Runtime task). Runs with a single task report as before, except that `air list --all-status` now orders runs by submission time, as the run index pages them, rather than by their latest attempt's start. ([#7632](https://github.com/databricks/cli/pull/7632))
1 change: 1 addition & 0 deletions acceptance/air/control-chars/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
"runs": [
{
"run_id": "123",
"task_key": "train",
"run_name": "qwen\u001b[2J-train",
"user": "user\u001b[31m@example.com",
"status": "SUCCESS",
Expand Down
4 changes: 4 additions & 0 deletions acceptance/air/get-ai-runtime/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
"ts": "[TIMESTAMP]",
"data": {
"run_id": "123",
"task_key": "train",
"status": "SUCCESS",
"started_at": "[TIMESTAMP]",
"duration_seconds": 12,
Expand Down Expand Up @@ -73,6 +74,7 @@ MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
"ts": "[TIMESTAMP]",
"data": {
"run_id": "123",
"task_key": "train",
"status": "SUCCESS",
"started_at": "[TIMESTAMP]",
"duration_seconds": 12,
Expand Down Expand Up @@ -142,6 +144,7 @@ MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
"ts": "[TIMESTAMP]",
"data": {
"run_id": "123",
"task_key": "train",
"status": "SUCCESS",
"started_at": "[TIMESTAMP]",
"duration_seconds": 12,
Expand Down Expand Up @@ -172,6 +175,7 @@ MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
"ts": "[TIMESTAMP]",
"data": {
"run_id": "123",
"task_key": "train",
"status": "SUCCESS",
"started_at": "[TIMESTAMP]",
"duration_seconds": 12,
Expand Down
1 change: 1 addition & 0 deletions acceptance/air/get-mlflow-retry/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
"ts": "[TIMESTAMP]",
"data": {
"run_id": "123",
"task_key": null,
"status": "FAILED",
"started_at": "[TIMESTAMP]",
"duration_seconds": 12,
Expand Down
1 change: 1 addition & 0 deletions acceptance/air/get-multitask-skipped/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
56 changes: 56 additions & 0 deletions acceptance/air/get-multitask-skipped/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@

=== get (text): the skipped AI Runtime task's own status, with no duration

>>> [CLI] air get 123

╭─ Metadata ─────────────────────────────────────────────────────╮
│ │
│ Run ID 123 │
│ Status ● UPSTREAM_FAILED │
│ Submitted N/A │
│ Retries 0 │
│ Max Retries 0 │
│ Duration N/A │
│ Experiment air-byot-multitask-dag │
│ MLflow Run N/A │
│ User user@example.com │
│ Accelerators 1x A10 │
│ Environment N/A │
│ │
╰────────────────────────────────────────────────────────────────╯

Run URL: [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]

=== get (json)

>>> [CLI] air get 123 -o json
{
"v": 1,
"ts": "[TIMESTAMP]",
"data": {
"run_id": "123",
"task_key": "train",
"status": "UPSTREAM_FAILED",
"started_at": null,
"duration_seconds": null,
"attempt_number": 0,
"experiment_name": "air-byot-multitask-dag",
"dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
"mlflow_url": null,
"mlflow_experiment_id": null,
"mlflow_run_id": null,
"est_remaining_seconds": null,
"est_percent_complete": null,
"user": "user@example.com",
"accelerator_type": "GPU_1xA10",
"num_accelerators": 1,
"environment_version": null,
"base_environment": null,
"max_retries": {
"limit": 0,
"source": "reported"
},
"mlflow_run_name": null,
"sweep": null
}
}
5 changes: 5 additions & 0 deletions acceptance/air/get-multitask-skipped/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
title "get (text): the skipped AI Runtime task's own status, with no duration"
trace $CLI air get 123

title "get (json)"
trace $CLI air get 123 -o json
53 changes: 53 additions & 0 deletions acceptance/air/get-multitask-skipped/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# The SDK occasionally probes host reachability with a HEAD request; stub it so
# the test is deterministic.
[[Server]]
Pattern = "HEAD /"
Response.Body = ''

# The notebook-first DAB shape where the prep notebook fails, so the
# ai_runtime_task that depends on it is skipped (UPSTREAM_FAILED) and never
# starts. `air get` must report the task's own UPSTREAM_FAILED, not the run's
# FAILED, and no start time or duration rather than the run's.
[[Server]]
Pattern = "GET /api/2.2/jobs/runs/get"
Response.Body = '''
{
"run_id": 123,
"run_page_url": "https://my-workspace.cloud.databricks.test/jobs/runs/123",
"creator_user_name": "user@example.com",
"start_time": 1700000000000,
"end_time": 1700000005000,
"state": {"life_cycle_state": "TERMINATED", "result_state": "FAILED", "state_message": "Task prep failed"},
"tasks": [
{
"task_key": "prep",
"run_id": 455,
"attempt_number": 0,
"start_time": 1700000000000,
"end_time": 1700000005000,
"state": {"life_cycle_state": "TERMINATED", "result_state": "FAILED", "state_message": "prep: table not found"},
"notebook_task": {"notebook_path": "/Workspace/Users/user@example.com/prep"}
},
{
"task_key": "train",
"run_id": 456,
"attempt_number": 0,
"max_retries": 0,
"state": {"life_cycle_state": "SKIPPED", "result_state": "UPSTREAM_FAILED", "state_message": "Upstream task(s) failed."},
"depends_on": [{"task_key": "prep"}],
"ai_runtime_task": {
"experiment": "air-byot-multitask-dag",
"deployments": [
{"compute": {"accelerator_type": "GPU_1xA10", "accelerator_count": 1}}
]
}
}
]
}
'''

[[Server]]
Pattern = "GET /api/2.2/jobs/runs/get-output"
Response.Body = '''
{}
'''
1 change: 1 addition & 0 deletions acceptance/air/get-multitask/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
142 changes: 142 additions & 0 deletions acceptance/air/get-multitask/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@

=== get (text): the first AI Runtime task, not the notebook prep task

>>> [CLI] air get 123

╭─ Metadata ─────────────────────────────────────────────────────╮
│ │
│ Run ID 123 │
│ Task train │
│ Status ● SUCCESS │
│ Submitted 2023-11-14 22:13 UTC │
│ Retries 0 │
│ Max Retries 0 │
│ Duration 10s │
│ Experiment air-byot-multitask-dag │
│ MLflow Run matmul-smoke │
│ User user@example.com │
│ Accelerators 1x A10 │
│ Environment 5 │
│ │
╰────────────────────────────────────────────────────────────────╯

Run URL: [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1

=== get --task-key eval (text)

>>> [CLI] air get 123 --task-key eval

╭─ Metadata ─────────────────────────────────────────────────────╮
│ │
│ Run ID 123 │
│ Task eval │
│ Status ● FAILED │
│ Submitted 2023-11-14 22:13 UTC │
│ Retries 0 │
│ Max Retries 2 │
│ Duration 8s │
│ Experiment eval-exp │
│ MLflow Run matmul-smoke │
│ User user@example.com │
│ Accelerators 8x H100 │
│ Environment 6 │
│ │
╰────────────────────────────────────────────────────────────────╯

Failure reason: eval: loss is NaN (step 120)

Run URL: [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1

=== get (json)

>>> [CLI] air get 123 -o json
{
"v": 1,
"ts": "[TIMESTAMP]",
"data": {
"run_id": "123",
"task_key": "train",
"status": "SUCCESS",
"started_at": "[TIMESTAMP]",
"duration_seconds": 10,
"attempt_number": 0,
"experiment_name": "air-byot-multitask-dag",
"dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
"mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
"mlflow_experiment_id": "exp1",
"mlflow_run_id": "run1",
"est_remaining_seconds": null,
"est_percent_complete": null,
"user": "user@example.com",
"accelerator_type": "GPU_1xA10",
"num_accelerators": 1,
"environment_version": "5",
"base_environment": null,
"max_retries": {
"limit": 0,
"source": "reported"
},
"mlflow_run_name": "matmul-smoke",
"sweep": null
}
}

=== get --task-key eval (json)

>>> [CLI] air get 123 --task-key eval -o json
{
"v": 1,
"ts": "[TIMESTAMP]",
"data": {
"run_id": "123",
"task_key": "eval",
"status": "FAILED",
"started_at": "[TIMESTAMP]",
"duration_seconds": 8,
"attempt_number": 0,
"experiment_name": "eval-exp",
"dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
"mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
"mlflow_experiment_id": "exp1",
"mlflow_run_id": "run1",
"termination_reason": "eval: loss is NaN (step 120)",
"est_remaining_seconds": null,
"est_percent_complete": null,
"user": "user@example.com",
"accelerator_type": "GPU_8xH100",
"num_accelerators": 8,
"environment_version": "6",
"base_environment": null,
"max_retries": {
"limit": 2,
"source": "reported"
},
"mlflow_run_name": "matmul-smoke",
"sweep": null
}
}

=== get --task-key prep: not an AI Runtime task

>>> [CLI] air get 123 --task-key prep
Error: run 123 has no AI Runtime task "prep"; its AI Runtime tasks are: train, eval

Exit code: 1

=== get --task-key prep (json)

>>> [CLI] air get 123 --task-key prep -o json
{
"v": 1,
"ts": "[TIMESTAMP]",
"error": {
"code": "INVALID_ARGS",
"kind": "PERMANENT",
"message": "run 123 has no AI Runtime task \"prep\"; its AI Runtime tasks are: train, eval",
"retryable": false
}
}

Exit code: 1
17 changes: 17 additions & 0 deletions acceptance/air/get-multitask/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
title "get (text): the first AI Runtime task, not the notebook prep task"
trace $CLI air get 123

title "get --task-key eval (text)"
trace $CLI air get 123 --task-key eval

title "get (json)"
trace $CLI air get 123 -o json

title "get --task-key eval (json)"
trace $CLI air get 123 --task-key eval -o json

title "get --task-key prep: not an AI Runtime task"
errcode trace $CLI air get 123 --task-key prep

title "get --task-key prep (json)"
errcode trace $CLI air get 123 --task-key prep -o json
Loading
Loading