diff --git a/docs/advanced/airflow.md b/docs/advanced/airflow.md index 5d00ed5..1d3cba1 100644 --- a/docs/advanced/airflow.md +++ b/docs/advanced/airflow.md @@ -251,7 +251,7 @@ A build here depends on these, but no DAG runs them, so no task waits for them: stream-raw (a Spark stream that fluid apply keeps running). ``` -A contract whose only builds are such streams gets `No build needs a DAG` from `fluid generate schedule`, not `No scheduler engine specified`, and no file is written. +A contract whose only builds are such streams gets `No build needs a DAG` from `fluid generate schedule`, not `No scheduler engine specified`, and no file is written. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A stream that declares `execution.retries` is the exception on Airflow: it gets a keepalive DAG, and a contract of such streams writes a file. See [Keeping a stream running](#keeping-a-stream-running). ### Dataset triggers @@ -438,10 +438,86 @@ The build phase of that apply starts one run on EMR Serverless, Glue, EMR on EC2 - **Timeouts.** `execution.runtime.timeout`, an ISO-8601 duration such as `PT2H`, is how long the run may take; six hours when unset. The task's timeout is the three hours every apply DAG gets plus that wait. A value that is not a duration of at least one second is refused before `tofu apply` changes anything (`apply_spark_invalid`). - **Retries** come from `execution.retries.maxAttempts`, as for every scheduled build. -- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A scheduler task can keep it running between applies: it runs `fluid apply --mode reconcile --build-id --yes` every few minutes, and starts the stream again when it died, within its restart budget. See [Reconcile mode](../cli/apply.md#reconcile-mode). A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers). +- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A scheduler task can keep a stream running between applies: it runs `fluid apply --mode reconcile --build-id --yes` every few minutes, and starts the stream again when it died, within its restart budget. See [Reconcile mode](../cli/apply.md#reconcile-mode). *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* `fluid generate schedule` writes that task, as a keepalive DAG, for a stream that declares `execution.retries`: see [Keeping a stream running](#keeping-a-stream-running). A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers). - **The worker runs the whole apply**, `tofu` included, so it needs the apply's permissions and the state backend, as for any scheduled build, plus `fluid` with boto3 (`data-product-forge[aws]` in MWAA's `requirements.txt`). The scheduler policy forge emits for a Spark product covers the run only. The operator settings (`FLUID_SPARK_*`) and the run's AWS actions are the ones [Spark to Iceberg on AWS](./spark-iceberg-aws.md#the-caller-s-permissions) lists. - A build that is not Spark gets the standard DAG, as before. A contract of dbt and Spark builds gets a DAG for each scheduled build. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* Builds that `builds[].dependencies` link share one DAG instead (see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers)). +### Keeping a stream running + +*(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A forge Spark stream that declares `execution.retries` gets a small DAG of its own, a keepalive. Its one task runs [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode) every few minutes, so a stream that a Spot interruption or a failed driver stopped starts again on its checkpoint, within the restart budget that `execution.retries` sets. + +- **Who gets one.** A Spark stream (a build with `properties.stream`) that is not `trigger.availableNow` and not `trigger.type: continuous`, and declares `execution.retries` as an object such as `{maxAttempts: 5}`, on Airflow, the default engine. It is opt-in: a stream without `execution.retries` gets no DAG, although `--mode reconcile` would restart it too (5 restarts per rolling hour when unset). A contract whose only builds are such streams gets this DAG, and no longer `No build needs a DAG`. Prefect, Dagster and [`fluid scaffold-composer`](../cli/scaffold-composer.md) write nothing for a stream, and their messages point at the Airflow keepalive (`--scheduler airflow`, or `fluid generate schedule`). A batch that depends on the stream does not wait for it: the stream is in no build graph. With no `orchestration.engine` in the contract, such a stream makes Airflow the default, as a build with a cron trigger does, and `fluid generate artifacts` (stage 3 of a pipeline) writes its DAG under `schedule/`. +- **What the DAG is.** A file `_dag.py`, with the `-` of the build id as `_`. Its DAG id is `[__]__`, and its one task is `fluid_keepalive`. The task runs, with the same `env -i` script and environment allowlist as every `fluid apply` task: + + ```text + cd "$FLUID_PROJECT_DIR" + fluid apply "$FLUID_PROJECT_DIR/$CONTRACT_PATH" --env "$FLUID_ENV_NAME" --mode reconcile --build-id "$BUILD_ID" --yes + ``` + + `--env` is left out when `FLUID_ENV_NAME` is empty. +- **Schedule and limits.** It runs every 5 minutes, `*/5 * * * *`, or on the build's `execution.orchestration.airflow.dagConfig.schedule`, which is validated like any DAG's cron. `@once` is refused, because a keepalive has to recur. It runs one run at a time (`max_active_runs=1`), with no catch-up, no Airflow retries and a 30-minute task timeout. The build's `execution.retries` is the restart budget (`maxAttempts` restarts per rolling hour, 5 when unset, at most 10), and `--mode reconcile` applies it. Each wait inside a run is bounded by [`FLUID_SPARK_WAIT_SECONDS`](./environment-variables.md#spark-on-aws-preview), 900 seconds by default. +- **No outlets.** A stream updates its tables all the time, and an asset event every 5 minutes would start every DAG scheduled on them. A dataset trigger on an expose that only the stream writes is refused, as before. +- **What a run does.** A healthy stream is a no-op. A stream that died starts again on its checkpoint. A run succeeds (exit `0`) when the stream is unchanged, restarted and still running, left stopped because it was cancelled on purpose, or there is nothing to reconcile. It fails (exit `1`, which Airflow's alerting picks up) when the stream drifted from the contract, nothing is deployed, the job config the last apply deployed cannot be read, the restarts are spent, EMR Serverless's or EMR on EKS's own retries are spent, the EMR on EC2 cluster ended, the restarted run ends at once, or an AWS call fails (a missing permission, say). Where forge restarts (EMR on EC2, Glue and a local build), a spent budget is not final: once the oldest restart is an hour old, the next run restarts the stream, and the runs before it fail. +- **Stopping it.** A stream cancelled on purpose stays stopped: `--mode reconcile` leaves a cancelled run alone. `fluid apply` starts it again. +- **The worker.** It needs the settings and the forge release that the CI apply has: the `FLUID_SPARK_*` operator settings (a missing allowlist refuses the build, and a setting that changes the compiled config reads as drift), and the role of the product's `fluid--spark-caller` policy, which lets it read the stream's job config and runner, and, on EMR on EC2, read the cluster and the step role. See [The caller's permissions](./spark-iceberg-aws.md#the-caller-s-permissions). + +For a stream with `execution.retries` and no other build, on Airflow: + +```text +Generated 1 files (airflow scheduler): + + dags/ingest_orders_dag.py + +Tip: To regenerate after editing the contract: fluid generate schedule +``` + +The DAG's docstring says what a run does: + +```text +Every run reconciles one always-on stream on the Airflow worker, with the +constants below filled in (`--env` is left out when FLUID_ENV_NAME is empty): + + cd "$FLUID_PROJECT_DIR" + fluid apply "$FLUID_PROJECT_DIR/$CONTRACT_PATH" --env "$FLUID_ENV_NAME" \ + --mode reconcile --build-id "$BUILD_ID" --yes + +It runs no tofu apply. When the stream died (a Spot interruption, a failed +driver), it starts again on its checkpoint, within the restart budget the +build's execution.retries sets, so the task itself never retries. The run +fails when the stream cannot be kept running: it drifted from the contract, +nothing is deployed, the job config the last apply deployed cannot be read, +its cluster ended, or the restarts are spent. It can also fail when this +worker's FLUID_SPARK_* settings or forge release differ from the CI apply's. +A run cancelled on purpose stays stopped until fluid apply starts it. It +runs on SCHEDULE, one run at a time, with no catch-up. The build's +execution.orchestration.airflow.dagConfig.schedule sets SCHEDULE. +``` + +With `--scheduler prefect` or `--scheduler dagster`, the same contract writes nothing: + +```text +No build needs a DAG: every build runs outside forge-cli (a Flink job, for example), as a stream that fluid apply keeps running, or continuously (trigger.type: continuous), so there is nothing to schedule. No files written. +A stream that declares execution.retries gets a keepalive DAG on Airflow: fluid generate schedule --scheduler airflow. +``` + +`fluid scaffold-composer`, which writes a Cloud Composer DAG, writes nothing for it either: + +```text +Every build runs outside forge-cli (a Flink job, for example), as a stream that +fluid apply keeps running, or continuously (trigger.type: continuous), so there +is nothing to schedule. No DAG written. +A stream that declares execution.retries gets a keepalive DAG from fluid +generate schedule, which restarts it when it stops. +``` + +`@once` as the schedule is refused: + +```text +generate_schedule_invalid_schedule: {'error': "'ingest-orders': execution.orchestration.airflow.dagConfig.schedule: a keepalive DAG reconciles the stream on a recurring cron, and @once is not one"} +``` + +These outputs are from the real `fluid generate schedule` and `fluid scaffold-composer` on `main`, on the example stream with `execution: {retries: {maxAttempts: 5}}` added; the DAG file's constants are `SCHEDULE = '*/5 * * * *'`, `RETRIES = 0` and a task timeout of 1800 seconds. No Airflow ran here: forge-cli's own tests load the DAG in a real Airflow 2 and Airflow 3 when one is named. + ## Cloud Composer deployment Upload a generated DAG to the environment's bucket: diff --git a/docs/advanced/production-troubleshooting.md b/docs/advanced/production-troubleshooting.md index 35de8ab..3e30434 100644 --- a/docs/advanced/production-troubleshooting.md +++ b/docs/advanced/production-troubleshooting.md @@ -231,7 +231,7 @@ The organization comes from `organization_id`, else `FLUID_CC_ORG_ID`; with neit |---|---|---| | `generated_path_outside_output_dir` | A contract field that becomes a file name (for example `builds[].properties.stages[].name`) contains a path separator or `..` | Rename the field | | `generated_path_collision` | Two generated files would resolve to the same path, so one would overwrite the other | Make the two names distinct | -| `generate_artifacts_skip_schedule_no_engine`, `..._engine_none`, `..._unreadable` | `fluid generate artifacts` skipped schedule artifacts because the contract has no `orchestration.engine` and no build with `execution.trigger.schedule`, the engine is `none`, or the file could not be read | Add the engine or a scheduled build if you want DAGs | +| `generate_artifacts_skip_schedule_no_engine`, `..._engine_none`, `..._unreadable` | `fluid generate artifacts` skipped schedule artifacts because the contract has no `orchestration.engine` and no build with `execution.trigger.schedule`, the engine is `none`, or the file could not be read | Add the engine or a scheduled build if you want DAGs. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A Spark stream that declares `execution.retries` counts as a scheduled build | | `generate_artifacts_failed` | `fluid generate artifacts` stopped on one generator. The `emit_key` line names it (`schedule`, for example, when `--env` is not a plain env name) | Read the `error` line and fix the input it names; see [`fluid generate artifacts`](../cli/generate-artifacts.md) | | `generate_schedule_invalid_schedule` (exit 2) | `fluid generate schedule` could not render the scheduled DAG from the contract's schedule or env | Read the `error` line; see [`fluid generate`](../cli/generate.md) | | `dbt_tests_refusing_overwrite` | `fluid generate dbt-tests --out` names an existing file that lacks fluid's managed-by header, so it is not overwritten | Delete the file, or pass a different `--out` | diff --git a/docs/advanced/spark-iceberg-aws.md b/docs/advanced/spark-iceberg-aws.md index 4b62606..4a5b254 100644 --- a/docs/advanced/spark-iceberg-aws.md +++ b/docs/advanced/spark-iceberg-aws.md @@ -1036,7 +1036,7 @@ Per runtime, the start is: ### The restart budget -*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A stream that died is started again on its checkpoint, within a budget of restarts per rolling hour. A plain `fluid apply` follows the rule, and so does [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode), which a keepalive runs. In 0.33.0 a stream whose last run ended `FAILED` was an error until the build changed, and `--mode reconcile` did not exist. +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A stream that died is started again on its checkpoint, within a budget of restarts per rolling hour. A plain `fluid apply` follows the rule, and so does [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode), which a keepalive runs. In 0.33.0 a stream whose last run ended `FAILED` was an error until the build changed, and `--mode reconcile` did not exist. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* `fluid generate schedule` writes a keepalive DAG for a stream that declares `execution.retries`: see [Keeping a stream running](./airflow.md#keeping-a-stream-running). - **The budget** is `execution.retries.maxAttempts` restarts per rolling hour: 5 when it is unset, and at most 10, the numbers that EMR Serverless applies to its own streaming retries. - **A death** is a run that ended `FAILED` or `INTERRUPTED` (an EMR step that died with its node), was lost with its machine, node or cluster, or was stopped by a signal from outside forge (a local run that was killed, or whose machine shut down), and a stream that stopped by itself. A local run whose supervisor got a signal from outside forge, a shutdown of the machine say, reads as failed now, and counts as a death: 0.33.0 read it as cancelled and started it again. A job killed on its own (`FAILED (exit -9)`) was failed in 0.33.0 too. @@ -1119,12 +1119,18 @@ The output above is from forge-cli's reconcile step, the code `fluid apply` runs *(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014))* A run's job config is an object in S3, which the runtime reads when the run starts and again each time it starts the job again after a failure. Until this change nothing checked what the object held: a config that another `fluid apply` had put there since this one compiled its runs would run under the config hash of this apply's runs, and a later `fluid apply` would take the hash for the config. Now a run starts only on the config that `fluid apply` compiled, in two steps. -1. **`fluid apply` reads the job config and the runner back from S3 before it starts a run, and starts nothing unless both are the ones it compiled.** This is for the runs on AWS: EMR Serverless, Glue, EMR on EC2 and EMR on EKS. The job config counts when its [digest](#the-digest) is the compiled config's. The runner counts when it is, byte for byte, the runner that this forge-cli ships. On EMR on EC2 the [check that the policy is attached](#an-emr-on-ec2-step-starts-once-its-policy-is-attached) comes first. A mismatch is the build's error, and `fluid apply` ends with exit code `1`. Nothing new was started. A running job of another config, which `fluid apply` cancels before it starts the new one, is already cancelled by then, so the stream stays stopped until the next `fluid apply`: +1. **`fluid apply` reads the job config and the runner back from S3 before it starts a run, and starts nothing unless both are the ones it compiled.** This is for the runs on AWS: EMR Serverless, Glue, EMR on EC2 and EMR on EKS. The job config counts when its [digest](#the-digest) is the compiled config's. The runner counts when it is, byte for byte, the runner that this forge-cli ships. A mismatch is the build's error, and `fluid apply` ends with exit code `1`. Nothing new was started. *(since 0.34.0, [forge-cli #1019](https://github.com/Agenticstiger/forge-cli/pull/1019))* The read-back comes before the cancel of a running job of another config, so that job keeps going, and the line says so. On EMR on EC2 the [check that the policy is attached](#an-emr-on-ec2-step-starts-once-its-policy-is-attached) is part of the start, which comes after that cancel: with the policy not attached, a job of another config is cancelled and nothing new starts. This is the line for a stream with no job running: ```text Spark ingest-orders (emr-ec2): failed. Spark build ingest-orders: s3://commerce-lake/_fluid/spark/commerce.orders-stream/ingest-orders/job.json or its runner is not the one this apply compiled (another apply has changed it since); nothing was started. Run fluid apply again: it uploads them and starts the run on them ``` + And this is the line for a stream whose running job has another config, `s-old` here, which the refusal leaves running: + + ```text + Spark ingest-orders (emr-ec2): refused on EMR cluster j-1ABCDEF. Spark build ingest-orders: s3://commerce-lake/_fluid/spark/commerce.orders-stream/ingest-orders/job.json or its runner is not the one this apply compiled (another apply has changed it since); nothing was started. Run fluid apply again: it uploads them and starts the run on them; run s-old was left running + ``` + An object that cannot be read, or is larger than 4 MiB, refuses the start in the same way. forge reads the config first and then the runner, and the message names the object it could not read: `s3://commerce-lake/_fluid/spark/commerce.orders-stream/ingest-orders/job.json cannot be read to check it before the run starts (AccessDenied); nothing was started`, or `... is larger than any artifact forge writes; nothing was started`. The word in parentheses is the code that AWS answered with, or the name of the error when AWS did not answer. A scheduler's start, `fluid apply --mode amend-and-build`, goes through the same check and fails in the same way. 2. **The run carries the digest, and the runner refuses any other config before it reads anything.** Every runtime, local included, starts the runner with `--fluid-job-sha256 ` after the arguments that identify the run. These are the runner's arguments in the EMR on EC2 step of the example above: @@ -1147,7 +1153,7 @@ The digest is the sha256 of the config's JSON in one canonical form: keys sorted #### What the check does not cover -The check guards against another apply changing the config or the runner. Write access to the artifact bucket stays the boundary: someone who can write there can replace the runner after the read. The identity that runs `fluid apply` needs `s3:GetObject` on the two objects, and `kms:Decrypt` where the bucket encrypts them with a KMS key. The scheduler's policy `fluid--spark-caller` already holds `s3:GetObject` on both, but not `kms:Decrypt`: add that for the scheduler's role where the bucket encrypts them with a KMS key. +The check guards against another apply changing the config or the runner. Write access to the artifact bucket stays the boundary: someone who can write there can replace the runner after the read. The identity that runs `fluid apply` needs `s3:GetObject` on the two objects, and `kms:Decrypt` where the bucket encrypts them with a KMS key. The scheduler's policy `fluid--spark-caller` holds `s3:GetObject` on both. *(since 0.34.0, [forge-cli #1019](https://github.com/Agenticstiger/forge-cli/pull/1019))* It holds `kms:Decrypt` for them too, through S3 only (`kms:ViaService`), so a scheduled start can read them from a bucket that encrypts them with a KMS key: see [The caller's permissions](#the-caller-s-permissions). The runner changed, so each running Spark stream restarts once on its next apply, on its checkpoint. @@ -1166,9 +1172,9 @@ The role that runs `fluid apply`, or the scheduler's execution role (the MWAA ro Scope `iam:PassRole` to the forge roles, so that a contract cannot pass a stronger one. -*(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014))* A start on AWS also reads the build's job config and runner from S3, so the role needs `s3:GetObject` on both objects: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). The generated `fluid--spark-caller` policy holds it in an `Artifacts` statement, and holds no `kms:Decrypt`. +*(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014))* A start on AWS also reads the build's job config and runner from S3, so the role needs `s3:GetObject` on both objects: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). The generated `fluid--spark-caller` policy holds it in an `Artifacts` statement. *(since 0.34.0, [forge-cli #1019](https://github.com/Agenticstiger/forge-cli/pull/1019))* Its `S3Kms` statement holds `kms:Decrypt` on any key, but only through S3 in the region (`kms:ViaService`), for those objects, or for their bucket under an S3 Bucket Key, so a bucket that encrypts them with a KMS key needs no IAM addition for the scheduler's role. The key's own policy must still allow the role, unless it leaves that to the account's IAM policies. -*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* The policy also covers the product's always-on streams: it lets a role start, watch and stop their runs and read their job config and runner, so that a keepalive can run [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode) from the scheduler's role (the MWAA role, say). It has one statement for each runtime, named `Run`, that lists the runtime's actions once for every target of the product's builds on it, one `PassRole` statement for each runtime that passes a role (EMR Serverless and EMR on EKS), and one `Artifacts` statement. Two more appear where they apply: `StepRoleEmrEc2`, the two IAM reads of an EMR on EC2 start, and `FindApplication`, `emr-serverless:ListApplications` on `*`, for an EMR Serverless build that names no `applicationId`. A managed policy holds at most 6,144 characters, and a statement for each build reached that limit at about six builds. Each build's target, role and artifacts are still named in them. A bucket that encrypts the artifacts with a KMS key also needs `kms:Decrypt` on its key for the scheduler's role, which forge does not grant. `--mode reconcile` reads the uploaded job config on every run, so a role that lacks `s3:GetObject` on it fails while the stream is healthy. +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* The policy also covers the product's always-on streams: it lets a role start, watch and stop their runs and read their job config and runner, so that a keepalive can run [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode) from the scheduler's role (the MWAA role, say). It has one statement for each runtime, named `Run`, that lists the runtime's actions once for every target of the product's builds on it, one `PassRole` statement for each runtime that passes a role (EMR Serverless and EMR on EKS), one `Artifacts` statement, and one `S3Kms` statement. Two more appear where they apply: `StepRoleEmrEc2`, the two IAM reads of an EMR on EC2 start, and `FindApplication`, `emr-serverless:ListApplications` on `*`, for an EMR Serverless build that names no `applicationId`. A managed policy holds at most 6,144 characters, and a statement for each build reached that limit at about six builds. Each build's target, role and artifacts are still named in them. `--mode reconcile` reads the uploaded job config on every run, so a role that lacks `s3:GetObject` on it fails while the stream is healthy. ### When a build leaves the contract diff --git a/docs/cli/apply.md b/docs/cli/apply.md index 99279bb..c07d6d2 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -169,6 +169,8 @@ Before 0.16.3 the flag was accepted and dropped, so a DDL-only apply exited 0. * fluid apply contract.fluid.yaml --env prod --mode reconcile --build-id ingest-orders --yes ``` +*(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* `fluid generate schedule` writes that task, as an Airflow DAG, for a stream that declares `execution.retries`: see [Keeping a stream running](../advanced/airflow.md#keeping-a-stream-running). + - **What it reads.** The contract, which goes through the same schema gate, `--build-id` check and apply-time hooks as any apply. A `plan.json` is refused with `apply_reconcile_needs_contract`. On AWS it also reads the job config that the last apply uploaded, from S3, on every run, so a missing `s3:GetObject` shows while the stream is healthy. It reads the runs from the runtime's own history, and keeps no state of its own. - **Which streams.** With `--build-id`, that one stream. A build that is not a forge Spark build, a batch, or a stream triggered `availableNow` is refused with `apply_reconcile_not_a_stream`: its scheduler runs it with `--mode amend-and-build --build-id `. Without `--build-id`, every always-on Spark stream of the contract. A contract with none prints `No always-on Spark stream in the contract: nothing to reconcile.` and exits `0`. - **What it never does.** It does not replace a config, and it does not stop a build the contract no longer holds: a full `fluid apply` does both. It cancels only a younger second copy of the running config. diff --git a/docs/cli/generate-artifacts.md b/docs/cli/generate-artifacts.md index 5d570dd..66a0367 100644 --- a/docs/cli/generate-artifacts.md +++ b/docs/cli/generate-artifacts.md @@ -79,7 +79,7 @@ When `schedule` is requested and the contract has nothing to schedule, the run l | Event | Cause | | --- | --- | -| `generate_artifacts_skip_schedule_no_engine` | No `orchestration.engine`, and no build with a schedule trigger. | +| `generate_artifacts_skip_schedule_no_engine` | No `orchestration.engine`, and no build with a schedule trigger. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A Spark stream that declares `execution.retries` counts as a build with a schedule, so it does not log this event. | | `generate_artifacts_skip_schedule_engine_none` | `orchestration.engine: none`. | | `generate_artifacts_skip_schedule_unreadable` | The contract file cannot be read at all. | @@ -115,7 +115,7 @@ dist/artifacts/schedule/__aws/ingest_subscriptions_dag.py The `schedule` key writes DAGs when either holds: - `orchestration.engine` is set to anything except `none`. -- No `orchestration.engine` is set, and a build declares `execution.trigger.schedule` (or `cron`) with trigger `type: schedule` or no type. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A `dataset` or `schedule_and_dataset` trigger counts too. The engine then defaults to Airflow. The file does nothing until `schedule-sync` delivers it, and stage 11 names the scheduler (`airflow`, `mwaa`, `composer` and `astronomer` run Airflow DAGs), so the default decides the file format only. +- No `orchestration.engine` is set, and a build declares `execution.trigger.schedule` (or `cron`) with trigger `type: schedule` or no type. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A `dataset` or `schedule_and_dataset` trigger counts too. The engine then defaults to Airflow. The file does nothing until `schedule-sync` delivers it, and stage 11 names the scheduler (`airflow`, `mwaa`, `composer` and `astronomer` run Airflow DAGs), so the default decides the file format only. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A forge Spark stream that declares `execution.retries` counts as such a build too: it gets a keepalive DAG, and the engine defaults to Airflow. Each such build gets its own DAG that runs `fluid apply`, unless the contract hand-declares `orchestration.tasks`, which keep their declared operators. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* Builds that `builds[].dependencies` link share one DAG, with a task for each build. A build is left out when its own `execution.orchestration.engine` is `none` or not Airflow, or when its trigger is `event`, `manual` or `timetable`, which a DAG on a cron or on datasets cannot express. A `dataset` or `schedule_and_dataset` build gets a DAG: see [Build graphs and dataset triggers](../advanced/airflow.md#build-graphs-and-dataset-triggers). In 0.32.0 and earlier, those two were left out too, and a contract with no `orchestration.engine` whose only trigger was `dataset` logged `generate_artifacts_skip_schedule_no_engine` and wrote no DAG. *(since 0.22.0, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* A `continuous` build, or one that runs outside forge-cli, gets no DAG and no task at all (see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)). diff --git a/docs/cli/generate.md b/docs/cli/generate.md index 045e3c2..66338a5 100644 --- a/docs/cli/generate.md +++ b/docs/cli/generate.md @@ -468,7 +468,7 @@ The DAG's contents, its worker requirements and the environment it passes to flu - A build with `trigger.type: dataset` gets a DAG that runs when the datasets in `trigger.datasets` are updated (all of them, or any one with `datasetsOperator: any`). `schedule_and_dataset` runs on its cron and on those updates. A dataset is an Airflow asset URI, or `{productId, exposeId}` of an expose of the contract. - Each task declares the exposes its build writes as `outlets`, so another DAG's dataset trigger can start on them. - A cycle, a dependency on the build itself or on no build, and the other graph errors stop generation with `generate_schedule_invalid_schedule` and exit 2. -- A contract whose only builds are streams that `fluid apply` keeps running prints `No build needs a DAG` and writes no files. In 0.32.0 and earlier it printed `No scheduler engine specified` when the contract set no engine. +- A contract whose only builds are streams that `fluid apply` keeps running prints `No build needs a DAG` and writes no files. In 0.32.0 and earlier it printed `No scheduler engine specified` when the contract set no engine. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A stream that declares `execution.retries` is the exception on Airflow: `generate schedule` writes it a keepalive DAG, whose one task runs `fluid apply --mode reconcile --build-id --yes` every 5 minutes. Prefect, Dagster and `fluid scaffold-composer` write nothing for it. See [Keeping a stream running](../advanced/airflow.md#keeping-a-stream-running). - With `--overwrite`, DAG files an earlier run wrote into `-o` for the same product and env that the contract no longer makes are removed. The [Airflow integration](../advanced/airflow.md#build-graphs-and-dataset-triggers) has a worked example, the asset URI of each platform and the errors. diff --git a/docs/cli/scaffold-composer.md b/docs/cli/scaffold-composer.md index 11a976e..4ddd378 100644 --- a/docs/cli/scaffold-composer.md +++ b/docs/cli/scaffold-composer.md @@ -65,5 +65,7 @@ with DAG( (trigger.type: continuous), so there is nothing to schedule. No DAG written. ``` + *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A stream that declares `execution.retries` has a keepalive DAG that this command does not write. The command adds a second line, `A stream that declares execution.retries gets a keepalive DAG from fluid generate schedule, which restarts it when it stops.`, and [`fluid generate schedule`](./generate.md#fluid-generate-schedule) writes the DAG: see [Keeping a stream running](../advanced/airflow.md#keeping-a-stream-running). + A DAG the command wrote for the product before stays where it is, and the command prints its path when it finds it under the name it writes now: delete it, and remove it from Cloud Composer if you deployed it there. A file of that name that it did not write is not mentioned. - For non-Composer Airflow installs, see [`fluid generate-airflow`](./generate-airflow.md). For Airflow, Dagster and Prefect output from one command, see [`fluid generate schedule`](./generate.md#fluid-generate-schedule). diff --git a/docs/cli/schedule-sync.md b/docs/cli/schedule-sync.md index fcf3e46..b3a25aa 100644 --- a/docs/cli/schedule-sync.md +++ b/docs/cli/schedule-sync.md @@ -164,7 +164,7 @@ The Jenkins and 7-system templates from `fluid generate ci` self-gate stage 11 o 1. **Reference-only contract**: stage 3 auto-skipped the schedule emitter. 2. **Stage 3 was toggled off**: there is no `dist/artifacts/` tree at all. -3. **Nothing to schedule**: the contract has no `orchestration.engine` and no build with a schedule trigger, so stage 3 emitted no DAG. +3. **Nothing to schedule**: the contract has no `orchestration.engine` and no build with a schedule trigger, so stage 3 emitted no DAG. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A Spark stream that declares `execution.retries` counts as a build with a schedule. Direct callers of `fluid schedule-sync` still get the strict hard-fail (`schedule_sync_dags_dir_missing`, exit `2`), so typos surface loudly. diff --git a/docs/upgrading.md b/docs/upgrading.md index 7e81752..f6cc5c8 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1470,9 +1470,9 @@ These changes are in 0.34.0. The pin on this site stays at 0.33.0 until the page - **The runner sets `failOnDataLoss` to `true` on every Kafka source.** That is Spark's default, so a stream whose offsets the topic no longer holds fails, as before, and does not skip records. - **Each running Spark stream restarts once on its next apply, on its checkpoint.** The runner changed, and its sha256 is part of the config hash, so the hash of a build whose contract did not change is another one. -*(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014))* +*(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014), [forge-cli #1019](https://github.com/Agenticstiger/forge-cli/pull/1019))* -- **A Spark run starts only on the job config that `fluid apply` compiled.** Every runtime, local included, passes the config's digest at submit (`--fluid-job-sha256`), and forge's runner refuses any other config before it reads anything. On AWS, `fluid apply` first reads the job config and the runner back from S3, and starts nothing unless they are the ones this apply compiled: the build fails with `or its runner is not the one this apply compiled`, and `fluid apply` ends with exit code `1`. A running job of another config is already cancelled by then, so run `fluid apply` again: it uploads both and starts the run. The identity that runs `fluid apply` needs `s3:GetObject` on both objects (the scheduler's `fluid--spark-caller` policy already holds `s3:GetObject`, but not `kms:Decrypt`). Write access to the artifact bucket stays the boundary. The runner changed, so each running Spark stream restarts once on its next apply, on its checkpoint. See [A run starts only on the config this apply compiled](./advanced/spark-iceberg-aws.md#a-run-starts-only-on-the-config-this-apply-compiled). +- **A Spark run starts only on the job config that `fluid apply` compiled.** Every runtime, local included, passes the config's digest at submit (`--fluid-job-sha256`), and forge's runner refuses any other config before it reads anything. On AWS, `fluid apply` first reads the job config and the runner back from S3, and starts nothing unless they are the ones this apply compiled: the build fails with `or its runner is not the one this apply compiled`, and `fluid apply` ends with exit code `1`. A running job of another config is not cancelled, and keeps going: run `fluid apply` again, and it uploads both and starts the run. The identity that runs `fluid apply` needs `s3:GetObject` on both objects, and `kms:Decrypt` where the bucket encrypts them with a KMS key (the scheduler's `fluid--spark-caller` policy holds both). Write access to the artifact bucket stays the boundary. The runner changed, so each running Spark stream restarts once on its next apply, on its checkpoint. See [A run starts only on the config this apply compiled](./advanced/spark-iceberg-aws.md#a-run-starts-only-on-the-config-this-apply-compiled). *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* @@ -1484,11 +1484,11 @@ These changes are in 0.34.0. The pin on this site stays at 0.33.0 until the page - **A Spark job on AWS can write its own table where Lake Formation governs new tables.** forge granted a job's role Lake Formation access to its table only when the output's binding declared a non-empty `governance.lakeFormation` block, so in an account that removed the `IAM_ALLOWED_PRINCIPALS` default for new tables, a stream without that block could not see its own table. The grant (`INSERT`, `ALTER`, `DESCRIBE`, `SELECT`, and `DELETE` for a batch job) is now emitted for such an output and for a stream's dead-letter table beside it, with a `count` that the plan decides from the account's settings: 1 where Lake Formation governs new tables, 0 where IAM alone decides, so an account that keeps the default creates nothing new. The identity that runs `fluid plan` and `fluid apply` needs `lakeformation:GetDataLakeSettings`. An output with a non-empty `governance.lakeFormation` block keeps its grants, and an EMR on EC2 job, which runs as the cluster's own role, gets none. See [Lake Formation writes](./advanced/spark-iceberg-aws.md#lake-formation-writes). -*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013), [forge-cli #1019](https://github.com/Agenticstiger/forge-cli/pull/1019))* - **`fluid apply --mode reconcile` keeps the Spark streams running and deploys nothing.** It runs no `tofu`, no DDL and no build, and it takes no plan. It starts a stream that died again on its checkpoint, within its restart budget, when what the last apply deployed is what the contract compiles to. When it is not, it starts nothing, exits `1` and names what to run. `--build-id ` picks one stream, and a keepalive, a scheduler task every few minutes, is what runs it. See [Reconcile mode](./cli/apply.md#reconcile-mode). - **A Spark stream that died starts again within a restart budget, on a plain `fluid apply` too.** On EMR on EC2, Glue and a local build, a stream whose last run ended `FAILED` was an error until the build changed. It is started again on its checkpoint now, up to `execution.retries.maxAttempts` times per rolling hour (5 when unset, at most 10), each restart carrying `--fluid-restart-of `, and a spent budget is the error. EMR Serverless and EMR on EKS keep their own retries, and forge starts nothing on top of them. A run cancelled through the runtime is left stopped by `--mode reconcile` and started again by a plain `fluid apply`. `execution.retries` is no longer part of the config hash on EMR on EC2 and Glue, so a stream there that declares it restarts once on its next apply. A local run whose supervisor got a signal from outside forge, a shutdown of the machine say, reads as failed, not cancelled. Of two copies of one config the older is kept, where the newer was. See [The restart budget](./advanced/spark-iceberg-aws.md#the-restart-budget). -- **The scheduler policy `fluid--spark-caller` covers always-on streams, and its statements change shape.** It lets a role start, watch and stop a stream's runs and read its job config and runner, so a keepalive can run on MWAA. It holds one statement for each runtime, with every build's target, role and artifacts, where it held one for each build, which reached the 6,144-character limit of a managed policy at about six builds. It holds no `kms:Decrypt`: a bucket that encrypts the artifacts with a KMS key needs it for the scheduler's role. See [The caller's permissions](./advanced/spark-iceberg-aws.md#the-caller-s-permissions). +- **The scheduler policy `fluid--spark-caller` covers always-on streams, and its statements change shape.** It lets a role start, watch and stop a stream's runs and read its job config and runner, so a keepalive can run on MWAA. It holds one statement for each runtime, with every build's target, role and artifacts, where it held one for each build, which reached the 6,144-character limit of a managed policy at about six builds. It holds `kms:Decrypt` through S3 only, for those artifacts, so a bucket that encrypts them with a KMS key needs no IAM addition for the scheduler's role (the key's own policy must still allow it, unless it leaves that to the account's IAM policies). See [The caller's permissions](./advanced/spark-iceberg-aws.md#the-caller-s-permissions). *(since 0.34.0, [forge-cli #1017](https://github.com/Agenticstiger/forge-cli/pull/1017))* @@ -1498,6 +1498,10 @@ These changes are in 0.34.0. The pin on this site stays at 0.33.0 until the page - **An Iceberg table on Glue plans no change after an engine commits to it.** forge emitted the table's location with a trailing slash. Iceberg keeps it without one and writes it back into Glue on every commit, so each `fluid apply` after a commit planned an update (`~1`) for each table that had been written, and sent an `UpdateTable` while a stream kept committing. The location is now emitted as Iceberg keeps it. The first apply after the upgrade updates once, in place, the location of each Iceberg table that Glue still holds with the trailing slash, and a table in a Hive format keeps its trailing slash. See [Iceberg tables in Glue](./providers/aws.md#iceberg-tables-in-glue). +*(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* + +- **`fluid generate schedule` writes a keepalive DAG for an always-on stream that declares `execution.retries`.** On Airflow, a Spark stream that is not `trigger.availableNow` and not `trigger.type: continuous` gets a DAG whose one task, `fluid_keepalive`, runs `fluid apply --mode reconcile --build-id --yes` every 5 minutes, or on the build's `execution.orchestration.airflow.dagConfig.schedule`. A stream that died starts again on its checkpoint, within the restart budget that `execution.retries` sets, and a run fails when the stream cannot be kept running. A stream that already declares `execution.retries` gets the DAG from its next `fluid generate schedule` or `fluid generate artifacts`. A stream without `execution.retries` gets no DAG, as before. A contract whose only builds are such streams gets this DAG instead of `No build needs a DAG`; Prefect, Dagster and `fluid scaffold-composer` still write nothing for it. The worker needs the `FLUID_SPARK_*` settings and the forge release that the CI apply has. See [Keeping a stream running](./advanced/airflow.md#keeping-a-stream-running). + ## Older releases Upgrading from before `0.15.0`? Start with the diff --git a/docs/walkthrough/11-stage-pipeline.md b/docs/walkthrough/11-stage-pipeline.md index 7d44c61..e121b9a 100644 --- a/docs/walkthrough/11-stage-pipeline.md +++ b/docs/walkthrough/11-stage-pipeline.md @@ -198,7 +198,7 @@ dist/artifacts/schedule/entertainment.genre_preferences_v1__dev/build_genre_pref - `schedule/__/_dag.py` is one Airflow DAG per scheduled build, one directory per product and environment. The `__` suffix is the `--env` you pass, or `$FLUID_ENV`. - `MANIFEST.json` holds a SHA-256 for each file and a Merkle root. -Stage 3 writes schedule artifacts when the contract sets `orchestration.engine` to anything but `none`, or when it sets no engine and a build declares `execution.trigger.schedule` (or `cron`) with trigger `type: schedule` or no type. With `orchestration.engine: none` it logs `generate_artifacts_skip_schedule_engine_none`; with neither an engine nor a scheduled build it logs `generate_artifacts_skip_schedule_no_engine`. Either way it writes no DAG and the stage succeeds. [Declarative Airflow](./airflow-declarative.md) shows what is in the DAG file. For a contract whose builds use `pattern: hybrid-reference`, `fluid generate ci` turns stage 3 off by default. +Stage 3 writes schedule artifacts when the contract sets `orchestration.engine` to anything but `none`, or when it sets no engine and a build declares `execution.trigger.schedule` (or `cron`) with trigger `type: schedule` or no type. With `orchestration.engine: none` it logs `generate_artifacts_skip_schedule_engine_none`; with neither an engine nor a scheduled build it logs `generate_artifacts_skip_schedule_no_engine`. Either way it writes no DAG and the stage succeeds. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* A forge Spark stream that declares `execution.retries` counts as a scheduled build: with no engine, stage 3 writes its keepalive DAG and logs nothing. [Declarative Airflow](./airflow-declarative.md) shows what is in the DAG file. For a contract whose builds use `pattern: hybrid-reference`, `fluid generate ci` turns stage 3 off by default. ::: warning Stage 3 reads the bundle, not a fragment root A contract split into fragments with `$ref` (the layout [`fluid split`](../cli/split.md) writes) is accepted by `validate`, `plan`, `diff` and `bundle`. As of 0.18.1, `fluid generate artifacts ` on such a contract fails without `--env` with `Provider error: FLUID expose has no usable name`, because it reads the unresolved YAML. The same command succeeds with `--env `, and on a stage-1 bundle, which is what the generated pipeline passes. diff --git a/docs/walkthrough/airflow-declarative.md b/docs/walkthrough/airflow-declarative.md index e798883..38841ff 100644 --- a/docs/walkthrough/airflow-declarative.md +++ b/docs/walkthrough/airflow-declarative.md @@ -137,7 +137,7 @@ The DAG is written for Airflow 3 (`airflow.sdk.DAG`, `BashOperator` from `apache ### Which builds get a DAG -Stage 3 (`fluid generate artifacts`) writes schedule artifacts when the contract sets `orchestration.engine` to anything but `none`, or when it sets no engine and a build declares `execution.trigger.schedule` (or `cron`) with `type: schedule` or no type. Airflow renders the second case. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A `dataset` or `schedule_and_dataset` trigger counts there too. +Stage 3 (`fluid generate artifacts`) writes schedule artifacts when the contract sets `orchestration.engine` to anything but `none`, or when it sets no engine and a build declares `execution.trigger.schedule` (or `cron`) with `type: schedule` or no type. Airflow renders the second case. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A `dataset` or `schedule_and_dataset` trigger counts there too. *(since 0.34.0, [forge-cli #1018](https://github.com/Agenticstiger/forge-cli/pull/1018))* So does a forge Spark stream that declares `execution.retries`: it gets a keepalive DAG. Within that, a build gets the `fluid apply` DAG unless its own `execution.orchestration.engine` is `none` or not Airflow, or its trigger is `event`, `dataset`, `schedule_and_dataset`, `manual` or `timetable`, which a cron DAG cannot express. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* `dataset` and `schedule_and_dataset` leave that list: those builds get a DAG too, and builds linked by `builds[].dependencies` share one (see [Build graphs and dataset triggers](../advanced/airflow.md#build-graphs-and-dataset-triggers)). *(since 0.22.0, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* A `continuous` build gets none either: its platform keeps it running (see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)). A contract that hand-declares `orchestration.tasks` keeps those operators instead. diff --git a/scripts/forge-cli-releases.json b/scripts/forge-cli-releases.json index 2767aac..744f01b 100644 --- a/scripts/forge-cli-releases.json +++ b/scripts/forge-cli-releases.json @@ -86,6 +86,8 @@ "1013": "0.34.0", "1014": "0.34.0", "1015": "0.34.0", - "1017": "0.34.0" + "1017": "0.34.0", + "1018": "0.34.0", + "1019": "0.34.0" } }