Skip to content

About

Recipes for building and running scalable machine learning workloads with Databricks AI Runtime.

Resources

Contributing

Security policy

Stars

20 stars

Watchers

0 watching

Forks

Repository files navigation

Databricks AI Runtime Cookbook

The Databricks AI Runtime Cookbook provides runnable examples for building and running machine learning workloads with Databricks AI Runtime.

Each recipe focuses on a specific use case and includes an AIR workload configuration together with the source code needed to run it. Use these recipes as starting points that you can inspect, run, and adapt for your own workloads.

Repository organization

Recipes are grouped into top-level directories by domain.

There are two exceptions:

  • Ray is an index of Ray recipes across the repository. Each recipe remains in its domain directory and uses a ray-core-, ray-data-, ray-train-, or ray-tune- prefix.
  • Serving contains end-to-end examples that train, register, and deploy a model. Other domain directories generally contain examples for a single training or inference workload.

Run a recipe

Each recipe is designed to run from its own directory using the same command.

Before running a recipe, set up the Databricks CLI for a workspace with access to AI Runtime:

  • Configure the Databricks CLI with AI Runtime (AWS | Azure).
  • Authenticate to your workspace with databricks auth login (AWS | Azure).
  • If you use a coding agent, install the AI Runtime agent skill: databricks aitools install --skills-only --skills databricks-ai-runtime (source).

Select a workspace profile

Select an existing workspace profile for the current shell, replacing <profile-name> with its name:

databricks auth profiles
export DATABRICKS_CONFIG_PROFILE="<profile-name>"
databricks auth describe

Check the host in auth describe before submitting a workload. Keep this variable set so submit, status, and log commands use the same workspace. See configuration profiles for profile setup.

Without a selected profile, the CLI uses the profile that databricks auth profiles marks as (Default). databricks auth login marks the first profile you create. databricks auth switch changes the mark. If no profile is marked, the CLI uses the DEFAULT profile if one exists.

To override this selection, pass --profile <profile-name>. Use the same override when monitoring that run:

databricks auth describe --profile <profile-name>
databricks air run -f workload.yaml --watch --profile <profile-name>

Submit and monitor

  1. cd into the recipe directory.
  2. Review workload.yaml for values that you need to configure. Recipes with secrets.HF_TOKEN require Hugging Face token and Databricks secret setup.
  3. Run the workload with databricks air run -f workload.yaml --watch.

For example:

cd getting-started/air-cli
databricks air run -f workload.yaml --watch

--watch streams logs until the run completes. To submit in the background, omit --watch, save the job run ID printed by air run, and use it to inspect the run or follow its logs:

databricks air run -f workload.yaml
RUN_ID="<job-run-id>"
databricks air get "$RUN_ID"
databricks air logs "$RUN_ID"

These commands use the selected profile. If you submitted with --profile, include the same flag on both monitoring commands. air logs follows an active run until completion and fetches logs for a completed run. Both monitoring commands also work after a watched command exits.

Contribute

See CONTRIBUTING.md for guidance on adding and validating recipes.

About

Recipes for building and running scalable machine learning workloads with Databricks AI Runtime.

Resources

Contributing

Security policy

Stars

20 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages