Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/air/help-runtime-env-vars.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* `air run -h config` now lists the environment variables AI Runtime sets on every node (such as `MLFLOW_RUN_ID` and `CODE_SOURCE_PATH`), and `air logs --help` notes that `--download-to` cannot be combined with `--tail` or `--minutes`. ([#7649](https://github.com/databricks/cli/pull/7649))
8 changes: 8 additions & 0 deletions acceptance/air/config-help/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,14 @@ config
root_path: .
command: python "$CODE_SOURCE_PATH/train.py"

Set on every node (use in command and training code):
CODE_SOURCE_PATH Extracted code_source directory
MLFLOW_RUN_ID MLflow run created for this job; log metrics and artifacts to it
NUM_NODES, NODE_RANK Node count and this node's rank (0 to NUM_NODES-1)
WORLD_SIZE, LOCAL_WORLD_SIZE GPUs across all nodes, and on this node
MASTER_ADDR, MASTER_PORT Rank-0 address for distributed launchers such as torchrun
HYPERPARAMETERS_PATH The parameters YAML, when parameters is set

Workload:
experiment_name (required) Name of the experiment.
compute (required) Which accelerators to run on and how many.
Expand Down
2 changes: 1 addition & 1 deletion acceptance/air/help/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ Examples:
databricks air logs 123456789 --download-to ./logs

Flags:
--download-to string Download all logs to this directory instead of printing
--download-to string Download all logs to this directory instead of printing; not combinable with --tail or --minutes
-h, --help help for logs
--minutes int Fetch only logs from the last N minutes
--node int Fetch logs from this node
Expand Down
2 changes: 1 addition & 1 deletion cmd/air/logs.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ JOB_RUN_ID is the job run ID returned by "databricks air run".`,
cmd.Flags().IntVar(&tail, "tail", 0, "Print the last N existing lines before following, or from a completed run (default 10000)")
cmd.Flags().IntVar(&minutes, "minutes", 0, "Fetch only logs from the last N minutes")
cmd.Flags().IntVar(&retry, "retry", -1, "View logs from a specific retry attempt; -1 means latest")
cmd.Flags().StringVar(&downloadTo, "download-to", "", "Download all logs to this directory instead of printing")
cmd.Flags().StringVar(&downloadTo, "download-to", "", "Download all logs to this directory instead of printing; not combinable with --tail or --minutes")
cmd.Flags().BoolVar(&review, "review", false, "Download logs from all nodes and filter for error signatures")
cmd.Flags().MarkHidden("review")

Expand Down
10 changes: 9 additions & 1 deletion cmd/air/runconfig.go
Original file line number Diff line number Diff line change
Expand Up @@ -667,7 +667,15 @@ func configSchema() configField {
accelerator_type: GPU_1xA10
code_source:
root_path: .
command: python "$CODE_SOURCE_PATH/train.py"`,
command: python "$CODE_SOURCE_PATH/train.py"

Set on every node (use in command and training code):
CODE_SOURCE_PATH Extracted code_source directory
MLFLOW_RUN_ID MLflow run created for this job; log metrics and artifacts to it
NUM_NODES, NODE_RANK Node count and this node's rank (0 to NUM_NODES-1)
WORLD_SIZE, LOCAL_WORLD_SIZE GPUs across all nodes, and on this node
MASTER_ADDR, MASTER_PORT Rank-0 address for distributed launchers such as torchrun
HYPERPARAMETERS_PATH The parameters YAML, when parameters is set`,
children: describeStruct(reflect.TypeFor[runConfig](), configHelpRoot),
}
}
Expand Down
Loading