Skip to content

air run: warn when code imports torch on the Standard environment - #7650

Closed
anshul0713 wants to merge 2 commits into
databricks:mainfrom
anshul0713:air-run-warn-missing-torch
Closed

anshul0713 wants to merge 2 commits into
databricks:mainfrom
anshul0713:air-run-warn-missing-torch

Conversation

@anshul0713

Copy link
Copy Markdown
Contributor

Changes

databricks air run, including --dry-run, now prints a warning before submitting in this case:

  • the code_source has a .py file that imports torch, and
  • the environment is the Standard environment at version 5 or later, which doesn't include PyTorch, and
  • torch isn't in environment.dependencies.
Warning: train.py imports torch, but environment version 6 (Standard) does not include PyTorch, so the run would fail after waiting for a GPU. Set environment.version: databricks_ai_v6, or add torch to environment.dependencies.

The check is skipped for unity_catalog_image, multi-container workloads, databricks_ai_v*, versions before 5, and dependency lists that reference a requirements file. It scans at most 200 .py files of up to 1 MiB each. It's only a hint and never blocks a submission.

Why

This config passes --dry-run, then fails with ModuleNotFoundError only after the run has waited for a GPU, and the default max_retries: 3 queues the same failure again. On the released v1.26.0, 2 of 3 agent trials of a 1x A10 PyTorch training task hit this, and one of them lost about 12 minutes waiting in the GPU queue before the failure showed up. Recent help text on main points to databricks_ai_v6, which makes this less common, but one trial still hit it on main. This warning is a local safety net that costs nothing.

Tests

  • New TestMissingTorchWarning, a table test. It warns for the default and Standard 6 environments, for import torch and from torch import, and when only other dependencies are listed. It stays quiet for databricks_ai_v6, a torch dependency, a requirements file, Standard 4, a custom image, and a commented-out import.
  • acceptance/air/run has two new cases. A Standard environment with import torch warns, and databricks_ai_v6 doesn't.
  • go test ./cmd/air/... and go test ./acceptance -run TestAccept/air pass.

This pull request and its description were written by Isaac.

The Standard environment has not included PyTorch since version 5. A
workload that imports torch there passes --dry-run, then fails only after
it waits for a GPU, and the default retries repeat the failure. Agents hit
this in most CAJ trials on the released CLI. Scan the code_source's .py
files and warn before submitting, pointing at databricks_ai_v6 or a torch
dependency.

Co-authored-by: Isaac <no-reply@databricks.com>
(cherry picked from commit 6d641a4658e301209c20136d4eb09303d4568125)
@anshul0713
anshul0713 requested review from a team as code owners November 16, 2026 04:23
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! This pull request was opened from a fork.

If you're a member of the Databricks organization, please request access to the cli-contrib team and open your pull request from a branch in this repository instead of a fork — fork PRs don't run integration tests automatically and have reduced CI access.

If you're an external contributor, no action is needed; a maintainer will review your pull request.

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 16, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 7650
  • Commit SHA: 4e852f895891b60dbad84528bb6a9134864af61d

Checks will be approved automatically on success.

@anshul0713

Copy link
Copy Markdown
Contributor Author

Superseded by #7656: the same change, opened from a branch in this repo so the full CI and integration tests can run.

@anshul0713 anshul0713 closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant