Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/advanced/airflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,7 @@ The build phase of that apply starts one run on EMR Serverless, Glue, EMR on EC2

- **Timeouts.** `execution.runtime.timeout`, an ISO-8601 duration such as `PT2H`, is how long the run may take; six hours when unset. The task's timeout is the three hours every apply DAG gets plus that wait. A value that is not a duration of at least one second is refused before `tofu apply` changes anything (`apply_spark_invalid`).
- **Retries** come from `execution.retries.maxAttempts`, as for every scheduled build.
- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers).
- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A scheduler task can keep it running between applies: it runs `fluid apply <contract> --mode reconcile --build-id <stream> --yes` every few minutes, and starts the stream again when it died, within its restart budget. See [Reconcile mode](../cli/apply.md#reconcile-mode). A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers).
- **The worker runs the whole apply**, `tofu` included, so it needs the apply's permissions and the state backend, as for any scheduled build, plus `fluid` with boto3 (`data-product-forge[aws]` in MWAA's `requirements.txt`). The scheduler policy forge emits for a Spark product covers the run only. The operator settings (`FLUID_SPARK_*`) and the run's AWS actions are the ones [Spark to Iceberg on AWS](./spark-iceberg-aws.md#the-caller-s-permissions) lists.
- A build that is not Spark gets the standard DAG, as before. A contract of dbt and Spark builds gets a DAG for each scheduled build. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* Builds that `builds[].dependencies` link share one DAG instead (see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers)).

Expand Down
Loading
Loading