Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/advanced/airflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ In practice:
- Set `FLUID_PROJECT_DIR` on the worker. Without it the task fails with `set FLUID_PROJECT_DIR on the Airflow worker to the directory that holds the product checkout`. The contract path is the path you gave the generator, relative to the working directory, or `--contract-path` when the worker's layout differs.
- Put the credentials the apply needs in the worker environment. The DAG starts `fluid` with an empty environment plus the names above, so a variable that is neither on the list nor named by a `{{ env.NAME }}` or `${NAME}` placeholder in the contract, nor an `env://NAME` `secretRef`, does not arrive. Add one with `FLUID_DAG_ENV_PASSTHROUGH="NAME1 NAME2"`.
- A DAG uses `max_active_runs=1` and `catchup=False`, retries from the build's `execution.retries.maxAttempts` (attempts minus one), and sets a three-hour execution timeout.
- *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* The task's `fluid apply` runs a dbt or Python build once, whichever of these its `trigger.type` is: `schedule`, `dataset`, `schedule_and_dataset`, `event`, `dependency` or `timetable`. The apply runs the build once. What decides when is the DAG for `schedule`, `dataset` and `schedule_and_dataset`, the DAG of the builds it depends on for `dependency`, and another scheduler or an event for `event` and `timetable`, which get no DAG. In 0.33.0 only `schedule` ran: a dbt or Python build of any other of these types failed with `Unknown trigger type` and exit code `1`, so the task of a `dataset` or `schedule_and_dataset` build, which this page generates a DAG for, failed. A `manual` build keeps its `iterations`, `continuous` is refused, and a `trigger.type` that is none of the names the schema knows fails with `Unknown trigger type` and the list `manual, dataset, dependency, event, schedule, schedule_and_dataset, timetable`. On 0.33.0 a `schedule_and_dataset` build printed this, and on `main` it runs once:
- *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* The task's `fluid apply` runs a dbt or Python build once, whichever of these its `trigger.type` is: `schedule`, `dataset`, `schedule_and_dataset`, `event`, `dependency` or `timetable`. The apply runs the build once. What decides when is the DAG for `schedule`, `dataset` and `schedule_and_dataset`, the DAG of the builds it depends on for `dependency`, and another scheduler or an event for `event` and `timetable`, which get no DAG. In 0.33.0 only `schedule` ran: a dbt or Python build of any other of these types failed with `Unknown trigger type` and exit code `1`, so the task of a `dataset` or `schedule_and_dataset` build, which this page generates a DAG for, failed. A `manual` build keeps its `iterations`, `continuous` is refused, and a `trigger.type` that is none of the names the schema knows fails with `Unknown trigger type` and the list `manual, dataset, dependency, event, schedule, schedule_and_dataset, timetable`. On 0.33.0 a `schedule_and_dataset` build printed this, and in 0.34.0 it runs once:

```text
Trigger: schedule_and_dataset
Expand Down Expand Up @@ -438,7 +438,7 @@ The build phase of that apply starts one run on EMR Serverless, Glue, EMR on EC2

- **Timeouts.** `execution.runtime.timeout`, an ISO-8601 duration such as `PT2H`, is how long the run may take; six hours when unset. The task's timeout is the three hours every apply DAG gets plus that wait. A value that is not a duration of at least one second is refused before `tofu apply` changes anything (`apply_spark_invalid`).
- **Retries** come from `execution.retries.maxAttempts`, as for every scheduled build.
- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A scheduler task can keep it running between applies: it runs `fluid apply <contract> --mode reconcile --build-id <stream> --yes` every few minutes, and starts the stream again when it died, within its restart budget. See [Reconcile mode](../cli/apply.md#reconcile-mode). A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers).
- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A scheduler task can keep it running between applies: it runs `fluid apply <contract> --mode reconcile --build-id <stream> --yes` every few minutes, and starts the stream again when it died, within its restart budget. See [Reconcile mode](../cli/apply.md#reconcile-mode). A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers).
- **The worker runs the whole apply**, `tofu` included, so it needs the apply's permissions and the state backend, as for any scheduled build, plus `fluid` with boto3 (`data-product-forge[aws]` in MWAA's `requirements.txt`). The scheduler policy forge emits for a Spark product covers the run only. The operator settings (`FLUID_SPARK_*`) and the run's AWS actions are the ones [Spark to Iceberg on AWS](./spark-iceberg-aws.md#the-caller-s-permissions) lists.
- A build that is not Spark gets the standard DAG, as before. A contract of dbt and Spark builds gets a DAG for each scheduled build. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* Builds that `builds[].dependencies` link share one DAG instead (see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers)).

Expand Down
8 changes: 4 additions & 4 deletions docs/advanced/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ fluid apply runtime/plan.json --bundle runtime/bundle.tgz --env aws --yes
| `FLUID_SPARK_EXECUTION_ROLE_ALLOWLIST` | The roles `parameters.spark.executionRoleArn` may name: a role ARN, or an ARN prefix ending in `/`. |
| `FLUID_SPARK_CONF_ALLOWLIST` | `sparkConf` keys beyond the built-in tuning keys: a key, or a prefix ending in `*`. A key that loads code, sets credentials or moves the catalog or checkpoint is refused even when listed. |
| `FLUID_SPARK_CONTRACT_SQL_ALLOWLIST` | The builds whose own `properties.sql` a Spark job may run: `<product>`, `<product>/<build>`, or `<product>/<build>@sha256:<digest>`. *(since 0.33.0, [forge-cli #990](https://github.com/Agenticstiger/forge-cli/pull/990))* `fluid validate` reads it too, and warns only about a build it does not approve: see [Contract SQL and `fluid validate`](./spark-iceberg-aws.md#contract-sql-and-fluid-validate). |
| `FLUID_SPARK_WAIT_SECONDS` | How long a reconcile waits for a cancelled run to stop and a started one to leave its pending states. Default 900. *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* It is also how long `fluid apply` waits, for each run, for the runs and the application that a destroy would leave going: see [When a build leaves the contract](./spark-iceberg-aws.md#when-a-build-leaves-the-contract). A value that is not a positive number is ignored. |
| `FLUID_SPARK_WAIT_SECONDS` | How long a reconcile waits for a cancelled run to stop and a started one to leave its pending states. Default 900. *(since 0.34.0, [forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008))* It is also how long `fluid apply` waits, for each run, for the runs and the application that a destroy would leave going: see [When a build leaves the contract](./spark-iceberg-aws.md#when-a-build-leaves-the-contract). A value that is not a positive number is ignored. |
| `FLUID_SPARK_MAVEN_REPOSITORY` | One https Maven mirror that replaces Confluent's repository in `spark.jars.repositories`. A URL with a user or password, a query, a comma, plain http or a link-local host is refused. |
| `FLUID_SPARK_GLUE_JARS_URL` | `s3://<bucket>/<prefix>/`: where the platform keeps the pinned jars of a Glue Confluent Avro build. |
| `FLUID_SPARK_SUBMIT_EXECUTABLE` | The `spark-submit` a build of `parameters.spark.runtime: local` starts: an absolute path to an executable outside the project, or a name on `PATH`. Default: `spark-submit` on `PATH`. *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* [Running a build on this machine](./spark-iceberg-aws.md#operator-settings-for-a-local-run). `fluid doctor --env` lists it. |
Expand Down Expand Up @@ -184,9 +184,9 @@ See [Airflow](./airflow.md) for which variables reach `fluid apply` on the worke
| `FLUID_DBT_FORWARD_ENV` | Extra environment variable names or prefixes the dbt runner forwards. |
| `DBT_EXECUTABLE` | The dbt command the runner and the `--dbt-validate` and `--dbt-tests-key auto` probes resolve first: an absolute path, a bare name on `PATH`, or a multi-token wrapper. Falls back to `dbt` on `PATH`, then the active venv. |
| `FLUID_DBT_TESTS_KEY` | `auto` (default), `tests` or `data_tests`: which YAML key generated dbt tests attach under. The `--dbt-tests-key` flag wins. |
| `DBT_DOCKER_IMAGE` | A prebuilt image for the dbt container fallback. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* It must be pinned by digest (`name[:tag]@sha256:<digest>`), and a tag is refused. See [Which dbt `fluid apply` runs](../cli/generate.md#which-dbt-fluid-apply-runs). |
| `DBT_BOOTSTRAP_IMAGE` | Base image the dbt container fallback installs dbt into. Default `python:3.12-slim`. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* The default is `python:3.12-slim` pinned by digest, and an override must be pinned by digest too. |
| `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<adapter><2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* Without it the install is the adapter's pinned package (`dbt-athena-community==1.11.1` for `athena`) plus `dbt-core==1.11.17`. It takes one or more exact requirements (`name==version`) separated by spaces or commas, and forge adds `dbt-core==1.11.17` unless the list names `dbt-core`: name it with the version you want to run another dbt-core. A range is refused. |
| `DBT_DOCKER_IMAGE` | A prebuilt image for the dbt container fallback. *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* It must be pinned by digest (`name[:tag]@sha256:<digest>`), and a tag is refused. See [Which dbt `fluid apply` runs](../cli/generate.md#which-dbt-fluid-apply-runs). |
| `DBT_BOOTSTRAP_IMAGE` | Base image the dbt container fallback installs dbt into. Default `python:3.12-slim`. *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* The default is `python:3.12-slim` pinned by digest, and an override must be pinned by digest too. |
| `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<adapter><2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* Without it the install is the adapter's pinned package (`dbt-athena-community==1.11.1` for `athena`) plus `dbt-core==1.11.17`. It takes one or more exact requirements (`name==version`) separated by spaces or commas, and forge adds `dbt-core==1.11.17` unless the list names `dbt-core`: name it with the version you want to run another dbt-core. A range is refused. |
| `FLUID_SOURCE_DATABASE`, `FLUID_SOURCE_SCHEMA` | Read through dbt's `env_var()` in generated `sources.yml`: when set, they supply a source's database and schema before the dbt target's own. |
| `FLUID_ROLLBACK_KEEP_LAST_N` | Per-product retention of `.fluid/rollback-state.json`. Default `20`. |
| `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. Emitting needs the `lineage` extra. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* |
Expand Down
Loading
Loading