From fd3ac56ca81262e5d611f67f9503c56a238b52f3 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 08:46:35 +0100 Subject: [PATCH] docs(dbt): fluid apply runs a build whatever its trigger, finds the dbt that has its adapter, and pins the container (forge-cli #1009) Which dbt apply runs, the pinned container and its refusals, the three variables, the trigger types, and the upgrade bullets. --- docs/advanced/airflow.md | 14 +++++++ docs/advanced/environment-variables.md | 6 +-- docs/cli/generate.md | 53 +++++++++++++++++++++++++- docs/upgrading.md | 6 +++ 4 files changed, 75 insertions(+), 4 deletions(-) diff --git a/docs/advanced/airflow.md b/docs/advanced/airflow.md index 145b59a..6c38845 100644 --- a/docs/advanced/airflow.md +++ b/docs/advanced/airflow.md @@ -78,6 +78,20 @@ In practice: - Set `FLUID_PROJECT_DIR` on the worker. Without it the task fails with `set FLUID_PROJECT_DIR on the Airflow worker to the directory that holds the product checkout`. The contract path is the path you gave the generator, relative to the working directory, or `--contract-path` when the worker's layout differs. - Put the credentials the apply needs in the worker environment. The DAG starts `fluid` with an empty environment plus the names above, so a variable that is neither on the list nor named by a `{{ env.NAME }}` or `${NAME}` placeholder in the contract, nor an `env://NAME` `secretRef`, does not arrive. Add one with `FLUID_DAG_ENV_PASSTHROUGH="NAME1 NAME2"`. - A DAG uses `max_active_runs=1` and `catchup=False`, retries from the build's `execution.retries.maxAttempts` (attempts minus one), and sets a three-hour execution timeout. +- *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* The task's `fluid apply` runs a dbt or Python build once, whichever of these its `trigger.type` is: `schedule`, `dataset`, `schedule_and_dataset`, `event`, `dependency` or `timetable`. The apply runs the build once. What decides when is the DAG for `schedule`, `dataset` and `schedule_and_dataset`, the DAG of the builds it depends on for `dependency`, and another scheduler or an event for `event` and `timetable`, which get no DAG. In 0.33.0 only `schedule` ran: a dbt or Python build of any other of these types failed with `Unknown trigger type` and exit code `1`, so the task of a `dataset` or `schedule_and_dataset` build, which this page generates a DAG for, failed. A `manual` build keeps its `iterations`, `continuous` is refused, and a `trigger.type` that is none of the names the schema knows fails with `Unknown trigger type` and the list `manual, dataset, dependency, event, schedule, schedule_and_dataset, timetable`. On 0.33.0 a `schedule_and_dataset` build printed this, and on `main` it runs once: + + ```text + Trigger: schedule_and_dataset + Command: dbt build + ❌ Unknown trigger type: schedule_and_dataset + Supported types: manual, schedule + ``` + + ```text + Trigger: schedule_and_dataset (manual apply override) + Command: dbt build + Iterations: 1 + ``` - The same product on two clouds needs a directory each (`` with no env, `__` with one), so each [schedule sync](#pushing-dags-to-the-scheduler) mirrors only its own DAGs. ## Build graphs and dataset triggers diff --git a/docs/advanced/environment-variables.md b/docs/advanced/environment-variables.md index 3b3f7fc..92f0945 100644 --- a/docs/advanced/environment-variables.md +++ b/docs/advanced/environment-variables.md @@ -184,9 +184,9 @@ See [Airflow](./airflow.md) for which variables reach `fluid apply` on the worke | `FLUID_DBT_FORWARD_ENV` | Extra environment variable names or prefixes the dbt runner forwards. | | `DBT_EXECUTABLE` | The dbt command the runner and the `--dbt-validate` and `--dbt-tests-key auto` probes resolve first: an absolute path, a bare name on `PATH`, or a multi-token wrapper. Falls back to `dbt` on `PATH`, then the active venv. | | `FLUID_DBT_TESTS_KEY` | `auto` (default), `tests` or `data_tests`: which YAML key generated dbt tests attach under. The `--dbt-tests-key` flag wins. | -| `DBT_DOCKER_IMAGE` | A prebuilt image for the dbt container fallback. | -| `DBT_BOOTSTRAP_IMAGE` | Base image the dbt container fallback installs dbt into. Default `python:3.12-slim`. | -| `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. | +| `DBT_DOCKER_IMAGE` | A prebuilt image for the dbt container fallback. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* It must be pinned by digest (`name[:tag]@sha256:`), and a tag is refused. See [Which dbt `fluid apply` runs](../cli/generate.md#which-dbt-fluid-apply-runs). | +| `DBT_BOOTSTRAP_IMAGE` | Base image the dbt container fallback installs dbt into. Default `python:3.12-slim`. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* The default is `python:3.12-slim` pinned by digest, and an override must be pinned by digest too. | +| `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* Without it the install is the adapter's pinned package (`dbt-athena-community==1.11.1` for `athena`) plus `dbt-core==1.11.17`. It takes one or more exact requirements (`name==version`) separated by spaces or commas, and forge adds `dbt-core==1.11.17` unless the list names `dbt-core`: name it with the version you want to run another dbt-core. A range is refused. | | `FLUID_SOURCE_DATABASE`, `FLUID_SOURCE_SCHEMA` | Read through dbt's `env_var()` in generated `sources.yml`: when set, they supply a source's database and schema before the dbt target's own. | | `FLUID_ROLLBACK_KEEP_LAST_N` | Per-product retention of `.fluid/rollback-state.json`. Default `20`. | | `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. Emitting needs the `lineage` extra. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* | diff --git a/docs/cli/generate.md b/docs/cli/generate.md index 5865f04..bfbef5f 100644 --- a/docs/cli/generate.md +++ b/docs/cli/generate.md @@ -239,7 +239,58 @@ Model `access` is written under `config:` whatever version is detected. If no dbt is detectable, the generator emits the older shapes. dbt v2 rejects those, so a CI job that generates without dbt installed and then runs dbt v2 fails at parse. Install the dbt you will run in the job that runs `generate`, or pass `--dbt-tests-key` explicitly for the tests key. forge-cli's source records the floors above as measured on dbt-core 1.8.10, 1.9.11, 1.10.0, 1.10.5, 1.10.8, 1.12.5 and 2.0.4. -When `fluid apply` runs a dbt build in a container, the fallback when the local dbt cannot run the build's adapter, the container installs `dbt-<2` and `dbt-core<2` into `python:3.12-slim`. Three variables change that: `DBT_BOOTSTRAP_IMAGE` replaces the base image, `DBT_ADAPTER_PACKAGE` replaces the install (and drops the `dbt-core<2` cap, which is how you opt into v2), and `DBT_DOCKER_IMAGE` runs a prebuilt image that already has dbt. +Up to 0.33.0, when `fluid apply` runs a dbt build in a container, the fallback when the local dbt cannot run the build's adapter, the container installs `dbt-<2` and `dbt-core<2` into `python:3.12-slim`. Three variables change that: `DBT_BOOTSTRAP_IMAGE` replaces the base image, `DBT_ADAPTER_PACKAGE` replaces the install (and drops the `dbt-core<2` cap, which is how you opt into v2), and `DBT_DOCKER_IMAGE` runs a prebuilt image that already has dbt. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* The fallback installs pinned packages into a pinned image, and the variables take pinned values only: see [Which dbt `fluid apply` runs](#which-dbt-fluid-apply-runs). + +#### Which dbt `fluid apply` runs + +*On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* `fluid apply` runs a dbt build with the dbt that `$DBT_EXECUTABLE` names, as it is and whatever adapter it has. Without `$DBT_EXECUTABLE`, it uses `dbt` on `PATH`, or the `dbt` next to the Python that runs forge, when that dbt has the build's adapter. Otherwise, with Docker available, it runs the build in a container. + +**The adapter.** The build's `engine` names it, as `dbt-snowflake` names `snowflake`. The engines `dbt` and `dbt-core` name none, so the build's `execution.runtime.platform` does: `snowflake`, `bigquery` (and `gcp`), `duckdb`, `redshift` (and `aws`), `postgres` (and `postgresql`), `spark`, `athena` (and `aws-athena`), `glue` (and `aws-glue`) and `databricks`. dbt lists an adapter under its plugin name, which can differ from its package's: `dbt-athena-community` installs the plugin `athena`. For dbt-core v1, forge matches the plugin name exactly against the names listed under `Plugins:` in `dbt --version`. A dbt Fusion (v2), which lists none, is used for any adapter. In 0.33.0 the engine `dbt-athena-community` was looked for as `athena-community`, so a dbt that had the Athena plugin, found without `$DBT_EXECUTABLE`, was never used, and the build went to a container. A build that names no adapter runs on the local dbt as it is. + +**The container.** With no local dbt that has the adapter, forge runs `docker run` on an image pinned by digest, installs the packages in it from wheels only, and runs dbt. This is the command that `fluid apply` prints for an Athena build, with the project's path shortened: + +```text +docker run --rm -v :/workspace/project -w /workspace/project -v :/workspace/profiles -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY python:3.12-slim@sha256:5e2bdb1b0d542d5039cffb8181c65930093949194a65c365dfb1b0fe13a4fa5b sh -lc PIP_CONFIG_FILE=/dev/null python -I -m pip install --isolated --index-url https://pypi.org/simple --quiet --no-cache-dir --disable-pip-version-check --only-binary=:all: dbt-core==1.11.17 dbt-athena-community==1.11.1 && dbt build --no-partial-parse --project-dir /workspace/project --profiles-dir /workspace/project --profile lake +``` + +The environment goes in by name (`-e NAME`), so no value is on the command line. pip runs isolated: `-I` keeps the mounted project off its path, the index is PyPI and no pip configuration file loads, and no `PIP_*` setting is passed into the container, so a project `.env` cannot point pip at another index. A `PYTHON*` setting reaches the container only when forge sets it itself or `FLUID_DBT_FORWARD_ENV` names it. forge pins dbt-core and one package for each of these adapters. Their dependencies resolve within those packages' own ranges, and nothing is hash-locked. + +| Adapter | Package pinned | +|---|---| +| `athena` | `dbt-athena-community==1.11.1` | +| `bigquery` | `dbt-bigquery==1.12.4` | +| `clickhouse` | `dbt-clickhouse==1.10.3` | +| `databricks` | `dbt-databricks==1.13.0` | +| `duckdb` | `dbt-duckdb==1.11.1` | +| `postgres` | `dbt-postgres==1.11.0` | +| `redshift` | `dbt-redshift==1.11.2` | +| `snowflake` | `dbt-snowflake==1.12.5` | +| `spark` | `dbt-spark==1.11.1` | +| `trino` | `dbt-trino==1.11.0` | + +dbt-core is pinned at `dbt-core==1.11.17`. An adapter that is not in the table, `glue` for one, runs in a container only when `DBT_ADAPTER_PACKAGE` or `DBT_DOCKER_IMAGE` is set. Without either, and once the AWS checks below pass, the build is refused with `forge has no pinned package for the dbt adapter 'glue', so it will not install one in a container. Install the adapter next to your dbt, or set DBT_ADAPTER_PACKAGE to an exact pin (for example dbt-==1.2.3)`. Without Docker, a local dbt that lacks the adapter refuses the build too: + +```text +❌ Unable to prepare dbt build 'marts': dbt executable '/venv/bin/dbt' has no 'athena' plugin. Install the adapter next to it (pip install 'dbt-athena-community==1.11.1'), set DBT_EXECUTABLE to a dbt that has it, or make Docker available. +``` + +**The variables.** Each takes only a pinned value, and refuses the rest before `docker` starts: + +- `DBT_BOOTSTRAP_IMAGE` is the base image that dbt is installed into, pinned by digest (`name[:tag]@sha256:<64 hex digits>`). A tag is refused: `DBT_BOOTSTRAP_IMAGE='python:3.12-slim' names a tag, and a tag can move: pin it by digest (python:3.12-slim@sha256:, from `docker buildx imagetools inspect python:3.12-slim`)`. +- `DBT_ADAPTER_PACKAGE` is one or more exact requirements (`name==version`), separated by spaces or commas. They replace the adapter's pin, and forge adds `dbt-core==1.11.17` unless the list names dbt-core, which is how a build runs another dbt-core, v2 included. A range is refused: `DBT_ADAPTER_PACKAGE must list exact pins (name==version), not 'dbt-glue>=1.0': the dbt container installs only pinned packages`. +- `DBT_DOCKER_IMAGE` is a prebuilt image that has dbt, pinned by digest. forge installs nothing in it. A tag is refused as for the base image. + +**AWS adapters.** The adapters `athena`, `glue` and `redshift` authenticate to AWS, and a container has no access to the host's AWS profiles: forge mounts the project and the profiles directory and nothing of `~/.aws`, and it does not pass in `AWS_PROFILE`, `AWS_DEFAULT_PROFILE`, `AWS_CONFIG_FILE`, `AWS_SHARED_CREDENTIALS_FILE`, `AWS_CA_BUNDLE` or the two token-file paths (`AWS_WEB_IDENTITY_TOKEN_FILE`, `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE`). Before `docker` starts, forge refuses a build that would find no credentials: when the dbt profile names an AWS profile (`aws_profile_name` or `iam_profile`), and when neither the environment (`AWS_ACCESS_KEY_ID` with `AWS_SECRET_ACCESS_KEY`, or the container credentials URI) nor the dbt profile holds keys. A Redshift build that does not use IAM needs none. The first message names the profile's field and never its value, and both give the two ways out: + +```text +❌ Unable to prepare dbt build 'marts': the local dbt has no 'athena' plugin, so this build would run in a container, and the dbt profile names an AWS profile (aws_profile_name), which a container cannot read. Install the adapter next to dbt (pip install 'dbt-athena-community==1.11.1'), or export that profile's short-lived keys (eval "$(aws configure export-credentials --profile --format env)") and run without naming a profile (unset AWS_PROFILE, or drop aws_profile_name from the dbt profile) +``` + +```text +❌ Unable to prepare dbt build 'marts': the local dbt has no 'athena' plugin, so this build would run in a container, and neither the environment nor the dbt profile holds AWS keys, and a container cannot read the host's AWS profiles. Install the adapter next to dbt (pip install 'dbt-athena-community==1.11.1'), or export short-lived keys (eval "$(aws configure export-credentials --format env)") +``` + +These outputs are from forge-cli's dbt runner, the code `fluid apply` calls for a dbt build, with a stand-in for the local dbt (its `--version` output) and for `docker`: no container ran and no AWS call was made. The project's path is ``. #### Mesh hub diff --git a/docs/upgrading.md b/docs/upgrading.md index e417f43..7f8c13a 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1470,6 +1470,12 @@ These changes are on forge-cli `main`. The pin on this site stays at 0.32.0 unti - **The runner sets `failOnDataLoss` to `true` on every Kafka source.** That is Spark's default, so a stream whose offsets the topic no longer holds fails, as before, and does not skip records. - **Each running Spark stream restarts once on its next apply, on its checkpoint.** The runner changed, and its sha256 is part of the config hash, so the hash of a build whose contract did not change is another one. +*On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* + +- **`fluid apply` runs a dbt or Python build whatever its trigger.** A dbt or Python build of type `dataset`, `schedule_and_dataset`, `event`, `dependency` or `timetable` failed under `fluid apply` with `Unknown trigger type`. That included the task of the DAG that `fluid generate schedule` renders for a `dataset` or `schedule_and_dataset` build (`fluid apply --build-id`). The dbt and Python runners now run such a build once. `manual` keeps its iterations, `continuous` is still refused, and a `trigger.type` that is none of the names the schema knows is refused. See [What the DAG does](./advanced/airflow.md#what-the-dag-does). +- **`fluid apply` finds the dbt that has the build's adapter.** An engine named after its package, `dbt-athena-community`, was looked for as `athena-community`, so a local dbt with the Athena plugin was never used. forge now asks for dbt's plugin name (`athena`), reads the adapter of an engine without one (`dbt`, `dbt-core`) from the build's platform, and matches the `Plugins:` list of `dbt --version` exactly. +- **The dbt container fallback is pinned, and refuses what it cannot pin or authenticate.** It runs `python:3.12-slim` pinned by digest and installs `dbt-core==1.11.17` and an exact adapter package from wheels only, with pip run isolated against PyPI. `DBT_ADAPTER_PACKAGE` must list exact pins, `DBT_DOCKER_IMAGE` and `DBT_BOOTSTRAP_IMAGE` must be pinned by digest, and an adapter with no pin of its own (`glue`, for one) runs in a container only with `DBT_ADAPTER_PACKAGE` or `DBT_DOCKER_IMAGE`. A build of an AWS adapter (`athena`, `glue`, `redshift`) whose dbt profile names an AWS profile, or of `athena`, `glue` or a Redshift IAM build that has no AWS keys in the environment or the dbt profile, is refused before `docker` starts, with the command that installs the adapter next to dbt or exports short-lived keys. forge mounts nothing of `~/.aws`. See [Which dbt `fluid apply` runs](./cli/generate.md#which-dbt-fluid-apply-runs). + ## Older releases Upgrading from before `0.15.0`? Start with the