diff --git a/docs/how-tos/airflow/README.md b/docs/how-tos/airflow/README.md index 4fee196..38b0622 100644 --- a/docs/how-tos/airflow/README.md +++ b/docs/how-tos/airflow/README.md @@ -2,7 +2,7 @@ title: "Airflow in Datacoves: Key Things to Know First" sidebar_label: What to know description: "Important notes before writing Airflow DAGs in Datacoves: Ruff linting, Datacoves decorators, and My Airflow for faster personal DAG development." -sidebar_position: 1 +sidebar_position: 10 id: airflow-index --- # Airflow - What to know diff --git a/docs/how-tos/airflow/airflow-3-migration-check.mdx b/docs/how-tos/airflow/airflow-3-migration-check.mdx new file mode 100644 index 0000000..8b902b6 --- /dev/null +++ b/docs/how-tos/airflow/airflow-3-migration-check.mdx @@ -0,0 +1,83 @@ +--- +title: Check your DAGs for Airflow 3 migration issues +sidebar_label: Airflow 3 migration check +description: "Use the datacoves airflow3-check command to find and auto-fix Airflow 2 to 3 migration issues in your DAG code before upgrading your environment." +sidebar_position: 20 +--- +# Check your DAGs for Airflow 3 migration issues + +:::info +The `datacoves airflow3-check` command is available in Datacoves 6.1 and later. It works regardless of the Airflow version your environment runs, so you can (and should) use it while still on Airflow 2. +::: + +When an environment is upgraded from Airflow 2 to Airflow 3, DAG code written for Airflow 2 may rely on imports, arguments, or behaviors that were removed or deprecated in Airflow 3, for example: + +| Airflow 2 pattern | Airflow 3 replacement | +| --- | --- | +| `from airflow.decorators import dag, task` | `from airflow.sdk import dag, task` | +| `from airflow.operators.bash import BashOperator` | `from airflow.providers.standard.operators.bash import BashOperator` | +| `schedule_interval="@daily"` | `schedule="@daily"` | +| `email_on_failure` / `email_on_retry` in `default_args` | SMTP notifications (`SmtpNotifier`) | +| Runtime-varying values in DAG definitions (e.g. `datetime.today()` in `start_date`) | Static values; Airflow 3 creates a new DAG version on every parse otherwise | + +Finding every occurrence across a repository by hand is tedious. The Datacoves code-server image ships the `datacoves airflow3-check` command, which scans your DAG code for all of these using [ruff's Airflow lint rules](https://docs.astral.sh/ruff/rules/#airflow-air), maintained alongside the Airflow project itself. + +## Run the check + +Open a terminal in your Datacoves workbench and run: + +```bash +datacoves airflow3-check +``` + +By default it checks the DAGs folder configured for your environment. Each finding shows the file, line, offending pattern, and the suggested replacement: + +``` +airflow3-suggested-update: `airflow.decorators.dag` is removed in Airflow 3.0 + --> orchestrate/dags/daily_run.py:13:2 +help: `dag` has been moved to `airflow.sdk` since Airflow 3.0 +``` + +The command exits with code `0` when no issues are found and `1` when there are findings, so you can also use it in scripts or CI. + +To check a different directory (for example a scratch folder, or a repo with a non-standard layout), pass it explicitly: + +```bash +datacoves airflow3-check path/to/dags +``` + +## Apply automatic fixes + +Some issues have unambiguous, safe fixes (for example `schedule_interval` -> `schedule`). Apply them with: + +```bash +datacoves airflow3-check --fix +``` + +Changes that could alter behavior, such as rewriting imports to their new locations, are reported but not applied automatically; review each finding's `help:` line and update the code by hand. + +:::tip +Commit or stash your work before running `--fix` so you can review the changes with `git diff`. +::: + +## Recommended workflow + +1. Run `datacoves airflow3-check` on your repository while the environment is still on Airflow 2 and work through the findings. Fixes like `schedule_interval` -> `schedule` already work on recent Airflow 2 releases; for new-location imports such as `airflow.sdk` (Airflow 3 only), use a portable pattern while both versions are in play: + + ```python + try: + # Airflow 3 + from airflow.sdk import dag, task + except ImportError: + # Airflow 2 + from airflow.decorators import dag, task + ``` + + The check recognizes this pattern and will not flag the Airflow 2 import inside the `except` branch. + +2. Re-run it until it reports no issues. +3. Coordinate the environment upgrade with your Datacoves administrator. + +:::note +The check covers Python DAG files. DAGs generated from YAML definitions (`dbt-coves generate airflow-dags`) are regenerated by the tooling and do not usually need manual migration. +::: diff --git a/docs/how-tos/airflow/api-triggered-dag.md b/docs/how-tos/airflow/api-triggered-dag.md index e4c8948..0c55a1d 100644 --- a/docs/how-tos/airflow/api-triggered-dag.md +++ b/docs/how-tos/airflow/api-triggered-dag.md @@ -2,7 +2,7 @@ title: How to Trigger an Airflow DAG Using Datasets sidebar_label: Trigger DAG via Datasets description: "Step-by-step guide to triggering Airflow DAGs via dataset events in Datacoves, enabling data-driven pipeline orchestration without time-based schedules." -sidebar_position: 6 +sidebar_position: 140 --- # How to Trigger a DAG using Datasets diff --git a/docs/how-tos/airflow/customize-worker-environment.md b/docs/how-tos/airflow/customize-worker-environment.md index 63e2aa3..e0b4b91 100644 --- a/docs/how-tos/airflow/customize-worker-environment.md +++ b/docs/how-tos/airflow/customize-worker-environment.md @@ -2,7 +2,7 @@ title: Set Up a Custom Environment for Airflow Workers sidebar_label: Worker - Custom Environment description: "Configure a custom Docker image for Airflow workers in Datacoves to include specific Python packages, libraries, or dependencies your DAGs require." -sidebar_position: 39 +sidebar_position: 150 --- # How to set up a custom environment for your Airflow workers diff --git a/docs/how-tos/airflow/dags/_category_.yaml b/docs/how-tos/airflow/dags/_category_.yaml index a9e6209..9c239e5 100644 --- a/docs/how-tos/airflow/dags/_category_.yaml +++ b/docs/how-tos/airflow/dags/_category_.yaml @@ -1,5 +1,5 @@ label: DAGs -position: 11 +position: 40 link: type: generated-index title: DAGs diff --git a/docs/how-tos/airflow/initial-setup.mdx b/docs/how-tos/airflow/initial-setup.mdx index bd93051..c23a6d9 100644 --- a/docs/how-tos/airflow/initial-setup.mdx +++ b/docs/how-tos/airflow/initial-setup.mdx @@ -2,7 +2,7 @@ title: Airflow Configuration Guide for Datacoves Admins sidebar_label: Initial Setup description: "Configure Airflow in Datacoves: set DAG paths, git sync branch, Kubernetes executor settings, and environment-level service connections for dbt jobs." -sidebar_position: 1 +sidebar_position: 50 --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; diff --git a/docs/how-tos/airflow/request-resources-on-workers.md b/docs/how-tos/airflow/request-resources-on-workers.md index 0ada008..e681f0a 100644 --- a/docs/how-tos/airflow/request-resources-on-workers.md +++ b/docs/how-tos/airflow/request-resources-on-workers.md @@ -2,7 +2,7 @@ title: Request Custom CPU & Memory for Airflow Workers sidebar_label: Worker - Request CPU & Memory description: "Configure Kubernetes resource requests for individual Airflow tasks in Datacoves to allocate specific CPU and memory limits per DAG task." -sidebar_position: 40 +sidebar_position: 160 --- # How to request more memory or cpu resources on a particular DAG task diff --git a/docs/how-tos/airflow/send-emails.md b/docs/how-tos/airflow/send-emails.md index 02eee0a..fbaf145 100644 --- a/docs/how-tos/airflow/send-emails.md +++ b/docs/how-tos/airflow/send-emails.md @@ -2,7 +2,7 @@ title: Send Email Notifications on Airflow DAG Failure sidebar_label: Notifications - Email description: "Configure email alerts for Airflow DAG failures in Datacoves: set up SMTP integration, define recipient lists, and handle task-level notifications." -sidebar_position: 34 +sidebar_position: 70 --- # How to send email notifications on DAG's failure diff --git a/docs/how-tos/airflow/send-ms-teams-notifications.md b/docs/how-tos/airflow/send-ms-teams-notifications.md index 6c5d717..9464ea0 100644 --- a/docs/how-tos/airflow/send-ms-teams-notifications.md +++ b/docs/how-tos/airflow/send-ms-teams-notifications.md @@ -2,7 +2,7 @@ title: Send MS Teams Alerts on Airflow DAG Failure sidebar_label: Notifications - MS Teams description: "Configure Microsoft Teams notifications for Airflow DAGs in Datacoves to receive alerts when pipeline tasks succeed, fail, or are retried." -sidebar_position: 35 +sidebar_position: 80 --- # How to send Microsoft Teams notifications on DAG's status diff --git a/docs/how-tos/airflow/send-slack-notifications.md b/docs/how-tos/airflow/send-slack-notifications.md index 0a84587..f1e9d54 100644 --- a/docs/how-tos/airflow/send-slack-notifications.md +++ b/docs/how-tos/airflow/send-slack-notifications.md @@ -2,7 +2,7 @@ title: Send Slack Notifications on Airflow DAG Status sidebar_label: Notifications - Slack description: "Set up Slack alerts for Airflow DAG events in Datacoves: connect a workspace, configure channel routing, and receive success or failure messages." -sidebar_position: 36 +sidebar_position: 90 --- # How to send Slack notifications on DAG's status diff --git a/docs/how-tos/airflow/sync-database.mdx b/docs/how-tos/airflow/sync-database.mdx index bb8f381..5ef6677 100644 --- a/docs/how-tos/airflow/sync-database.mdx +++ b/docs/how-tos/airflow/sync-database.mdx @@ -2,7 +2,7 @@ title: Sync the Internal Airflow Database in Datacoves sidebar_label: Sync Internal Database description: "How to sync the internal Airflow metadata database in Datacoves to apply configuration changes, reset state, or recover from database inconsistencies." -sidebar_position: 3 +sidebar_position: 130 --- # Sync Airflow database diff --git a/docs/how-tos/airflow/use-airflow-api.md b/docs/how-tos/airflow/use-airflow-api.md index 03a3019..9925ea0 100644 --- a/docs/how-tos/airflow/use-airflow-api.md +++ b/docs/how-tos/airflow/use-airflow-api.md @@ -2,7 +2,7 @@ title: Access the Airflow REST API in Datacoves sidebar_label: Airflow REST API description: "Authenticate with and call the Airflow REST API in Datacoves: generate API tokens, trigger DAGs, check run status, and query task instances programmatically." -sidebar_position: 2 +sidebar_position: 30 --- # How to use the Airflow API diff --git a/docs/how-tos/airflow/use-aws-secrets-manager.mdx b/docs/how-tos/airflow/use-aws-secrets-manager.mdx index 9f466b0..55f58ce 100644 --- a/docs/how-tos/airflow/use-aws-secrets-manager.mdx +++ b/docs/how-tos/airflow/use-aws-secrets-manager.mdx @@ -2,7 +2,7 @@ title: Use AWS Secrets Manager for Airflow in Datacoves sidebar_label: Secrets - AWS Secrets Manager description: "Configure AWS Secrets Manager as the secrets backend for Apache Airflow in Datacoves to securely store and retrieve connections and variables." -sidebar_position: 36 +sidebar_position: 100 --- # How to use AWS Secrets Manager in Airflow diff --git a/docs/how-tos/airflow/use-azure-key-vault.mdx b/docs/how-tos/airflow/use-azure-key-vault.mdx index fc7fac1..32e6c60 100644 --- a/docs/how-tos/airflow/use-azure-key-vault.mdx +++ b/docs/how-tos/airflow/use-azure-key-vault.mdx @@ -2,7 +2,7 @@ title: Use Azure Key Vault for Airflow in Datacoves sidebar_label: Secrets - Azure Key Vault description: "Configure Azure Key Vault as the secrets backend for Apache Airflow in Datacoves to securely store and retrieve connections and variables using Managed Identity." -sidebar_position: 37 +sidebar_position: 110 --- # How to use Azure Key Vault in Airflow diff --git a/docs/how-tos/airflow/use-datacoves-secrets-manager.mdx b/docs/how-tos/airflow/use-datacoves-secrets-manager.mdx index 531216d..ded97b3 100644 --- a/docs/how-tos/airflow/use-datacoves-secrets-manager.mdx +++ b/docs/how-tos/airflow/use-datacoves-secrets-manager.mdx @@ -2,7 +2,7 @@ title: Use the Datacoves Secrets Manager in Airflow sidebar_label: Secrets - Datacoves Manager description: "How to use the built-in Datacoves Secrets Manager to store and inject secrets into Airflow connections and variables without exposing credentials in code." -sidebar_position: 38 +sidebar_position: 120 --- # How to use Datacoves Secrets Manager in Airflow diff --git a/docs/how-tos/airflow/use-key-pair-authentication.md b/docs/how-tos/airflow/use-key-pair-authentication.md index 8f17a3a..ebbab5a 100644 --- a/docs/how-tos/airflow/use-key-pair-authentication.md +++ b/docs/how-tos/airflow/use-key-pair-authentication.md @@ -2,7 +2,7 @@ title: Use Key-Pair Authentication in Airflow sidebar_label: Key-Pair Authentication description: "Configure Snowflake key-pair authentication for Airflow connections in Datacoves instead of password-based credentials for improved security." -sidebar_position: 10 +sidebar_position: 60 --- # Using Key-Pair Authentication in Airflow