Repository navigation
air: find the AI Runtime task anywhere in a multi-task job in air list and air get - #7632
Conversation
Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 11 changed tests (limit=20, 0 not selected) 11 tests: regression=11 unreleased=0 coverage=0 failing=0
TestAccept/air/control-chars: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌main (749c68f): latest (v1.26.0): TestAccept/air/get: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌main (749c68f): latest (v1.26.0): TestAccept/air/get-ai-runtime: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌main (749c68f): latest (v1.26.0): TestAccept/air/get-mlflow-retry: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌main (749c68f): latest (v1.26.0): TestAccept/air/get-multitask: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌main (749c68f): latest (v1.26.0): TestAccept/air/get-multitask-skipped: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌main (749c68f): latest (v1.26.0): TestAccept/air/get-pending: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌main (749c68f): latest (v1.26.0): TestAccept/air/get-run-state: branch ✅ | main (749c68f) ❌ | latest (v1.26.0) ❌main (749c68f): latest (v1.26.0): 3 more failure details omitted, see the full report in the run. |
✅ Integration test reportCommit: bf8d601
Top 4 slowest tests (at least 2 minutes):
|
Co-authored-by: Isaac <no-reply@databricks.com>
52dfc19 to
06ba7d8
Compare
Co-authored-by: Isaac <no-reply@databricks.com>
06ba7d8 to
efdde9c
Compare
Co-authored-by: Isaac <no-reply@databricks.com>
efdde9c to
f7bb7ad
Compare
Address review on #7632: - A run narrowed to one AI Runtime task now reports that task's latest attempt's state, so a succeeded train next to a failed eval, or an AI Runtime task skipped after a failed prep notebook, no longer shows the run's aggregate. While the run is still active after a failed attempt, the run's state stands, since the task may be waiting to retry. - `--limit` counts runs on both the scan and index paths. - `task_key` is null for a run without an AI Runtime task, matching `air list`. - The `air list` logs pane notes when its logs belong to another task than the selected row's. - `cancel --all` previews every AI Runtime task of each run, and its help says a cancel stops the whole run. - Tests for per-task status with differing results, the run-counted limit, `cancel --all` on a multi-task run, and the logs note. Co-authored-by: Isaac <no-reply@databricks.com>
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
Thanks for turning Maggie's comments around so quickly. The status, --limit and task_key fixes look right. Before merge, I'd like to see the task lookups, the key check and the per-run grouping each written once. Details inline. Agreed that per-task air logs can be a follow-up.
Address review on #7632: - `airTaskKeys` is the one scan for AI Runtime tasks; `isAirRun` is built on it, and readers take `Tasks[0]` of the run narrowed by `taskRun` (`leadTask`), so `airTask` and `firstAirTaskIndex` are gone. `isAirTask` unwraps a foreach once (`taskSpec`) instead of checking twice. Callers outside `get` and `list` that read an unnarrowed run go through `defaultTaskRun`. - `selectAirTask` checks the key once with early returns; `taskRun` no longer re-checks it. The "no AI Runtime tasks" error no longer repeats. - `air get` and the detail pane share `taskView`. - Every list path passes runs as groups (`runGroup`): `--limit` counts groups, the index path sorts groups so a run's rows stay together, filtering happens in `matchingGroup`, the cache stores and reads a run's entries in one place, and `cancel --all` takes each group's first row and task keys. `countRuns` and `firstRowPerRun` are gone. Co-authored-by: Isaac <no-reply@databricks.com>
|
@pardis-beikzadeh-db thanks. a081563 writes each of these once:
Details are in each thread. Per-task |
…st` and `air get` Multi-task AIR jobs, such as the official DAB example that runs a notebook prep task before its ai_runtime_task, were omitted from `air list` and showed N/A experiment and accelerators in `air get`, because both read tasks[0]. - Discover the first AIR task in task order: an ai_runtime_task, a legacy gen_ai_compute_task with a training script, or a foreach wrapping one. Non-AIR tasks no longer feed status, timing or the MLflow lookup. - A job with several AIR tasks lists one row per task. Rows share the run_id; JSON output gains task_key. Each row reads its own task runs. - `air get` gains --task-key (default: the first AIR task), its JSON gains task_key, and the text view adds a Task line when the run has several. - The list detail pane opens the selected row's task; `cancel --all` cancels a multi-row run once. - The index-path row cache stores one row per AIR task (air-list-runs-v3). Verified against a real run of universe ai-compute/dabs/examples/03_multitask_dag on e2-dogfood (run 47834518189346). Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Address review on #7632: - A run narrowed to one AI Runtime task now reports that task's latest attempt's state, so a succeeded train next to a failed eval, or an AI Runtime task skipped after a failed prep notebook, no longer shows the run's aggregate. While the run is still active after a failed attempt, the run's state stands, since the task may be waiting to retry. - `--limit` counts runs on both the scan and index paths. - `task_key` is null for a run without an AI Runtime task, matching `air list`. - The `air list` logs pane notes when its logs belong to another task than the selected row's. - `cancel --all` previews every AI Runtime task of each run, and its help says a cancel stops the whole run. - Tests for per-task status with differing results, the run-counted limit, `cancel --all` on a multi-task run, and the logs note. Co-authored-by: Isaac <no-reply@databricks.com>
Address review on #7632: - `airTaskKeys` is the one scan for AI Runtime tasks; `isAirRun` is built on it, and readers take `Tasks[0]` of the run narrowed by `taskRun` (`leadTask`), so `airTask` and `firstAirTaskIndex` are gone. `isAirTask` unwraps a foreach once (`taskSpec`) instead of checking twice. Callers outside `get` and `list` that read an unnarrowed run go through `defaultTaskRun`. - `selectAirTask` checks the key once with early returns; `taskRun` no longer re-checks it. The "no AI Runtime tasks" error no longer repeats. - `air get` and the detail pane share `taskView`. - Every list path passes runs as groups (`runGroup`): `--limit` counts groups, the index path sorts groups so a run's rows stay together, filtering happens in `matchingGroup`, the cache stores and reads a run's entries in one place, and `cancel --all` takes each group's first row and task keys. `countRuns` and `firstRowPerRun` are gone. Co-authored-by: Isaac <no-reply@databricks.com>
a081563 to
2e0e1d5
Compare
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
Thanks, the lookups and grouping read much better now. Two things before merge:
-
+1 to @rclarey's threads, especially
"runs": nullon an empty list and the state swap on single-task runs. Those change whatair runusers see today. -
The acceptance outputs don't show either change, because no fixture has them. Could we add acceptance cases for:
air list -o jsonwith no matching runsair geton a single-task run whose run state differs from its task's (e.g. run TIMEDOUT, task SUCCESS)air listtext output for a run with two AI Runtime tasks
Related: most of the new Go tests call helpers directly (
taskRun,selectAirTask,cancelPreviewRows, ...), which is why the last commit had to rewrite eight of them after helpers were renamed. Once the acceptance cases cover the behavior, those can go.
Two small nits inline.
…vironment, timing and nodes Review follow-ups for multi-task AI Runtime jobs: - A run whose only task is the AI Runtime task keeps its run-level state, status and timing, so single-task `air run` submissions report as before (termination prefix, run-level TIMEDOUT). - A skipped task (upstream failed, upstream cancelled, excluded) keeps its own state, and a task of a multi-task run reports only its own timing, never the whole run's. - Only a pending run, or a running run whose task waits to start, reports as waiting for compute; queued, blocked and waiting-for-retry runs report that state, as on main. - `air get` reads the environment keyed by the selected task's environment_key instead of `default`. - `air logs --download-to` and `--node` validation read the node count from the task the logs come from. - `air list` names the task in the Run ID column for a run with several AI Runtime tasks, reports `task_key` as null without one (matching `air get`), prints `"runs": []` for an empty list, and sorts a run by its first started task. - The list fixture comment no longer implies the default scan reaches bundle-deployed (JOB_RUN) jobs. Co-authored-by: Isaac <no-reply@databricks.com>
- acceptance/air/get-run-state: a single-task run that TIMEDOUT after its task succeeded reports the run's TIMEDOUT state and termination message. - acceptance/air/list: a run with two AI Runtime tasks lists as two rows that name their task and report their own status, and `air list -o json` with no matching runs prints "runs": []. - Drop the unit tests these and the existing get-multitask and cancel cases now cover, which called helpers directly. - selectAirTask starts from defaultTaskRun for an empty key, and taskRun drops a guard its callers make unnecessary. Co-authored-by: Isaac <no-reply@databricks.com>
|
@pardis-beikzadeh-db, thanks. Both points are addressed:
On the helper-level tests: I dropped the ones these cases and the existing I kept the tests for behavior that would need many more fixtures to cover end to end:
If you'd rather have any of these as acceptance cases instead, I can convert them. |
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
Thanks, @rclarey's threads, my two nits and the three acceptance cases all look done. A few non-blocking follow-ups inline.
…rough firstTask Follow-ups from review: - downloadLogs resolves the retry once and passes that task run to resolveNodeCount; logTaskRun becomes a plain lookup by task run ID, and runLogs's --node check goes through nodeCountForRetry. - firstTask's synthetic foreach task carries EnvironmentKey, so taskEnvironmentKey reads firstTask(run).EnvironmentKey instead of unwrapping the sweep a third time. - acceptance/air/get-multitask-skipped: prep fails and train is skipped, so the golden shows UPSTREAM_FAILED and no start time or duration. - The run-level waiting-state test asserts the exact status. Co-authored-by: Isaac <no-reply@databricks.com>
…eued task as pending Review follow-ups: - The index path keeps a batch in the index's order (newest submission first) instead of re-sorting by row start time, which a task that hasn't started or was skipped no longer has. Such a run now lists where main lists it, fresh or cached. - A task of a multi-task run that is pending, queued or waiting to retry reports PENDING with the waiting-for-compute message, as a single-task run does; a task blocked on an upstream task still reports BLOCKED. - The per-task state cases that had a single task key get a sibling task, so they reach taskState, and get-multitask carries status.termination_details at run and task level, so the task's own failure reason is pinned. - The changelog and `air get` help state that a task whose latest attempt failed reads RUNNING while another task of the run is still active. Co-authored-by: Isaac <no-reply@databricks.com>
Review follow-ups: - Only an attempt Jobs may retry (failed, timed out, internal error) makes a task of an active run read the run's RUNNING; a cancelled task, like a skipped one, keeps its own state. - Tests pin the index path's order: a newer active run lists above cached older ones, and a retried run lists by submission time. The changelog notes that `air list --all-status` now orders runs by submission time. Co-authored-by: Isaac <no-reply@databricks.com>
❌ Integration test reportCommit: 562948c
522 interesting tests: 490 MISS, 14 KNOWN, 9 RECOVERED, 5 PANIC, 4 FAIL
Top 50 slowest tests (at least 2 minutes):
|
Changes
Multi-task AIR jobs are supported, and the official DAB example (universe
ai-compute/dabs/examples/03_multitask_dag) runs a notebookpreptask before itsai_runtime_task.air listandair getreadtasks[0], soair list --all-statuslisted that run with null experiment/accelerators andair getrendered it withN/AExperiment and Accelerators. (A bundle-deployed job runs as aJOB_RUN, which only--all-statusreaches, through the run index; the defaultair list,--all-usersandcancel --allscanSUBMIT_RUNruns only, and this PR doesn't change that.)Selection rule:
ai_runtime_task, a legacygen_ai_compute_taskwith a training script, or afor_each_taskwhose inner task is one. A run is an AIR run iff it has at least one such task. Non-AIR tasks (the prep notebook) no longer feed status, timing, experiment, accelerators or the MLflow lookup: the run is narrowed to the selected task's task runs (all attempts) before the display helpers read it.air listshows one row per AIR task. Rows sharerun_id, and the text table names the task in the Run ID column (123 train); JSON gainstask_key(nullfor a run without an AIR task, as inair get). Each row's fields and MLflow link come from its own task runs, including its own status and timing (a skipped task keeps its state and reports no timing), and the experiment/accelerator filters apply per task. A run whose only task is the AIR task keeps its run-level state and timing, so single-taskair runoutput is unchanged. The index-path row cache stores the whole set per run (air-list-runs-v3).cancel --allcancels a multi-row run once.air get. New--task-keyflag; default is the first AIR task. A key that is not one of the run's AIR tasks is anINVALID_ARGSerror listing them. JSON gainstask_key; the text view adds aTaskline only when the run has several AIR tasks. The Environment cell reads the selected task'senvironment_key. Theair listdetail pane opens the selected row's task.for_each_taskwrapping an AIR task is that AIR task (sweep view /is_sweep). A job mixing a direct AIR task and a foreach AIR task now yields one row (or one selectable task) each, andair listandair getagree on the default (today list inspectedtasks[0]while get picked any foreach).Out of scope:
air logskeeps its last-task behavior;--download-toand--nodevalidation now read the node count from that same task.Why
Follows the single-task fix in #7587.
Tests
Unit tests for the five requested shapes (AIR task first, AIR task second, two AIR tasks, foreach AIR task, no AIR task) plus retries sharing a key, run narrowing, per-task list rows and filters,
air get --task-keyselection and error, the detail pane, the legacygen_ai_compute_taskpath and the per-task row cache. New acceptance testsacceptance/air/get-multitask(per-task status, environment and--task-keyerrors)acceptance/air/get-multitask-skipped(a task skipped becauseprepfailed reports UPSTREAM_FAILED and no timing) andacceptance/air/get-run-state(a single-task run keeps its run-level TIMEDOUT state). Theair listfixture gains the notebook-first DAB shape, a run with two AI Runtime tasks (one row per task, named in the Run ID column), and an empty-o jsonlist.Verified against a real run of
03_multitask_dagdeployed to e2-dogfood (run47834518189346) with a build of this branch and a build ofmain:mainair list --all-status -o jsonexperiment_name: null,accelerator_type: nullair-byot-multitask-dag,GPU_1xA10×1,task_key: trainair getN/A, AcceleratorsN/Aair-byot-multitask-dag,1x A10air get --task-key preprun … has no AI Runtime task "prep"; its AI Runtime tasks are: traingo test ./cmd/air,go test ./acceptance -run TestAccept/airandgolangci-lint run ./cmd/air/...pass.This pull request and its description were written by Isaac.