diff --git a/docs/advanced/airflow.md b/docs/advanced/airflow.md index 6c38845..36aed1b 100644 --- a/docs/advanced/airflow.md +++ b/docs/advanced/airflow.md @@ -438,7 +438,7 @@ The build phase of that apply starts one run on EMR Serverless, Glue, EMR on EC2 - **Timeouts.** `execution.runtime.timeout`, an ISO-8601 duration such as `PT2H`, is how long the run may take; six hours when unset. The task's timeout is the three hours every apply DAG gets plus that wait. A value that is not a duration of at least one second is refused before `tofu apply` changes anything (`apply_spark_invalid`). - **Retries** come from `execution.retries.maxAttempts`, as for every scheduled build. -- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers). +- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A scheduler task can keep it running between applies: it runs `fluid apply --mode reconcile --build-id --yes` every few minutes, and starts the stream again when it died, within its restart budget. See [Reconcile mode](../cli/apply.md#reconcile-mode). A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers). - **The worker runs the whole apply**, `tofu` included, so it needs the apply's permissions and the state backend, as for any scheduled build, plus `fluid` with boto3 (`data-product-forge[aws]` in MWAA's `requirements.txt`). The scheduler policy forge emits for a Spark product covers the run only. The operator settings (`FLUID_SPARK_*`) and the run's AWS actions are the ones [Spark to Iceberg on AWS](./spark-iceberg-aws.md#the-caller-s-permissions) lists. - A build that is not Spark gets the standard DAG, as before. A contract of dbt and Spark builds gets a DAG for each scheduled build. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* Builds that `builds[].dependencies` link share one DAG instead (see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers)). diff --git a/docs/advanced/spark-iceberg-aws.md b/docs/advanced/spark-iceberg-aws.md index 3ecbb72..3a3fdcf 100644 --- a/docs/advanced/spark-iceberg-aws.md +++ b/docs/advanced/spark-iceberg-aws.md @@ -346,7 +346,7 @@ With `alertOn: [schema_violation]`, the job writes a line like this one to its s {"forge_spark_runner": "dead_letter_alert", "reason": "schema_violation", "records": 600} ``` -`fluid apply` treats the stopped job as a failed run, like any other (on EMR Serverless, after the retries that [the reconcile table](#starting-and-reconciling-runs) describes). With the same config hash it does not start the job again. It names the run and its state, and the message says: `It is not started again with the same config, which would fail the same way: fix the cause and change the build, and fluid apply starts the new config`. A change to the build is a change to the config hash, and a new `maxRecordsBeforeAbort` is one: `fluid plan` writes a new `configHash` for it in the plan's `spark.reconcile` action. The next `fluid apply` then starts the job on the same checkpoint. +`fluid apply` treats the stopped job as a failed run, like any other (on EMR Serverless, after the retries that [the reconcile table](#starting-and-reconciling-runs) describes). With the same config hash it does not start the job again. It names the run and its state, and the message says: `It is not started again with the same config, which would fail the same way: fix the cause and change the build, and fluid apply starts the new config`. A change to the build is a change to the config hash, and a new `maxRecordsBeforeAbort` is one: `fluid plan` writes a new `configHash` for it in the plan's `spark.reconcile` action. The next `fluid apply` then starts the job on the same checkpoint. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* On EMR on EC2, Glue and a local build, a job that stopped at the limit is a stream that died, and `fluid apply` and `--mode reconcile` start it again on its checkpoint within the [restart budget](#the-restart-budget). The count belongs to the job, so it starts at 0 again in each restart, and the budget bounds the restarts. #### Masked builds @@ -601,7 +601,7 @@ A mapped column can set `maskAtIngest: hash`. The column is the HMAC-SHA256 of i | Logs | A CloudWatch log group `/aws/fluid/spark//`, encrypted with a per-product KMS key that only CloudWatch Logs may use, for this product's groups. | | The job's role | Unless the contract names one. Least privilege: S3 read and write on the data and checkpoint prefixes, read on the artifacts, the Glue calls on the output, KMS through S3 and Secrets Manager only, MSK IAM on the cluster, topic and consumer group, `lakeformation:GetDataAccess`, the mask key and a registry secret, and its log group. | | A dead-letter table | *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* For a stream with `deadLetter` and `sink: table`: an Iceberg v2 `aws_glue_catalog_table` beside the output's table, with its S3 prefix, Glue table and, when Lake Formation governs the output, Lake Formation permissions added to the job's role. *On forge-cli `main`, not yet in a release ([forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010)).* Without a `governance.lakeFormation` block, those permissions are added where the account's Lake Formation settings govern new tables: see [A job's grant on its own table](#a-job-s-grant-on-its-own-table). A `sink: topic` emits no table: on MSK with IAM the job's role gets the permissions to write the topic. See [What OpenTofu emits for dead letters](#what-opentofu-emits-for-dead-letters). | -| A caller policy | For a build that runs to completion (batch, or a stream with `availableNow`): the managed policy `fluid--spark-caller`, for the platform to attach to the scheduler's role. | +| A caller policy | For a build that runs to completion (batch, or a stream with `availableNow`): the managed policy `fluid--spark-caller`, for the platform to attach to the scheduler's role. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A stream that runs continuously gets it too, so that a keepalive can run `fluid apply --mode reconcile` from the scheduler's role. | Per runtime: @@ -882,6 +882,8 @@ A local stream follows the `spark.reconcile` action of [Starting and reconciling | A stream that failed with this config | An error that names the run and its log, and no new start. Change the build, and apply starts the new config. | | A stream of a build that the contract no longer declares | Stops it: `stopped run CANCELLED`. A build moved to an AWS runtime keeps its local runs, because an apply of AWS builds never stops them. | +*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* The rows about a stream stopped from outside forge and a stream that failed change. A stream that failed, one whose supervisor got a signal from outside forge (a shutdown of the machine, say), and one whose processes ended without recording how (`ended LOST`) are a death: `fluid apply` and `--mode reconcile` start it again on its checkpoint within the [restart budget](#the-restart-budget), and count each restart. A spent budget is the error. Only a run that was cancelled through forge is left stopped, and only by `--mode reconcile`. + A stop sends SIGTERM to the run's process group, and SIGKILL after a grace period of 60 seconds. forge waits for the run to end for as long as `FLUID_SPARK_WAIT_SECONDS` says. A new topic or cluster on a checkpoint that an earlier run read is refused before anything is stopped, as on AWS. A run whose processes ended without recording how, because the machine restarted or something killed them with SIGKILL, did not fail. The next apply starts the same config again and says that the run stopped without recording its end (forge-cli's tests). @@ -902,6 +904,8 @@ Spark ingest-orders (local): failed run 20261119T060909Z-89c3a970 FAILED on this The two lines of that message, `the stream failed` and the exit code, come from a stand-in `spark-submit` that exits 1; the path is shortened to ``. +*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A failed stream is started again on its checkpoint, within the [restart budget](#the-restart-budget), and a message like the one above is the error for a spent budget. It ends with the same second way: `when the cause was this machine's (spark-submit, its properties file), remove /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs (the build's run history here) and fluid apply starts it again`. The run directory it names is the whole history, which is what the budget is counted from. + **A dry run.** `fluid apply --dry-run` starts and stops nothing and creates no run directory. It prints `would be reconciled to config ` for a stream, and `would stop ` for a run the contract no longer runs here: ```bash @@ -1015,8 +1019,8 @@ After `tofu apply`, `fluid apply` reconciles each streaming build: | A running job with this hash | Nothing. | | A running job with another hash | Cancels it, waits for it to stop, and starts the new config on the **same checkpoint**, so it resumes where the old one stopped. | | A new topic or cluster, with a checkpoint an earlier run read | Refuses, before cancelling anything. A checkpoint holds the offsets of one topic on one cluster. To read the new stream, move `properties.stream.checkpoint` to a new prefix: the job then starts from `startingOffsets`. | -| No running job | Starts one, unless the last run with this hash failed. | -| A last run that failed | An error that names the run and its reason. On EMR Serverless a streaming job retries until it fails more than `maxFailedAttemptsPerHour` times in an hour, and then stops, so starting it again would fail the same way. Fix the cause, then change the build: `fluid apply` starts it with the new config. | +| No running job | Starts one, unless the last run with this hash failed. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* Unless it failed where the service's own retries are spent (EMR Serverless and EMR on EKS) or the [restart budget](#the-restart-budget) is spent: see the next row. | +| A last run that failed | An error that names the run and its reason. On EMR Serverless a streaming job retries until it fails more than `maxFailedAttemptsPerHour` times in an hour, and then stops, so starting it again would fail the same way. Fix the cause, then change the build: `fluid apply` starts it with the new config. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* On EMR on EC2, Glue and a local build the stream is started again on its checkpoint within the [restart budget](#the-restart-budget), and the error is for a spent budget. EMR Serverless and EMR on EKS keep the error, as their own retries come first. | | A second active copy of the current config | Cancels it. | | A stream moved to `trigger.availableNow` | Cancels its old continuous job. Its scheduler starts it from now on. | @@ -1030,11 +1034,72 @@ Per runtime, the start is: - **EMR on EKS:** `StartJobRun` on `virtualClusterId`. - **Local:** *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* `spark-submit --deploy-mode client` on this machine, under a detached supervisor: see [Running a build on this machine](#running-a-build-on-this-machine). +### The restart budget + +*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A stream that died is started again on its checkpoint, within a budget of restarts per rolling hour. A plain `fluid apply` follows the rule, and so does [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode), which a keepalive runs. In 0.33.0 a stream whose last run ended `FAILED` was an error until the build changed, and `--mode reconcile` did not exist. + +- **The budget** is `execution.retries.maxAttempts` restarts per rolling hour: 5 when it is unset, and at most 10, the numbers that EMR Serverless applies to its own streaming retries. +- **A death** is a run that ended `FAILED` or `INTERRUPTED` (an EMR step that died with its node), was lost with its machine, node or cluster, or was stopped by a signal from outside forge (a local run that was killed, or whose machine shut down), and a stream that stopped by itself. A local run whose supervisor got a signal from outside forge, a shutdown of the machine say, reads as failed now, and counts as a death: 0.33.0 read it as cancelled and started it again. A job killed on its own (`FAILED (exit -9)`) was failed in 0.33.0 too. +- **A cancel is not a death.** A run that was cancelled through the runtime (`aws emr cancel-steps`, `CancelJobRun`, `BatchStopJobRun`, or forge's own cancel of a local run) was stopped on purpose. `--mode reconcile` leaves it stopped, and a plain `fluid apply` starts it again, as the operator who applies asks, without counting a restart. To stop a stream on purpose, cancel its run: `--mode reconcile` leaves it stopped, and a plain `fluid apply` starts it again. A step that a terminating cluster cancelled while it was pending is not an operator's stop: the reconcile reports the ended cluster. +- **Who restarts.** forge restarts a stream on EMR on EC2, on Glue, whose job forge emits with `MaxRetries` 0, and on a local build: none of them has a service that retries. EMR Serverless and EMR on EKS retry a failed run themselves, from `execution.retries`. A run that ended `FAILED` there has spent those retries, and forge starts nothing on top of them; a stream there that stopped without failing restarts within the budget. +- **The count.** Each restart carries `--fluid-restart-of ` in the runner's arguments, where the EMR console shows it in the step's arguments, and the budget is counted from the runtime's own run history (the `ListSteps` of EMR on EC2 reaches back an hour). forge keeps no state. A restart that was cancelled afterwards does not count. +- **A spent budget** starts nothing and is an error. It names the budget, the restarts and when the next one is allowed, and it is not final: once fewer restarts than the budget are under an hour old, the next apply or reconcile restarts the stream. To start it sooner, fix the cause and change the build, and `fluid apply` starts the new config. +- **`execution.retries` is no longer part of the config hash on EMR on EC2 and Glue**, and a local build has none: none of them takes a retry setting, so a new budget restarts nothing. A stream there that already declares `execution.retries` restarts once on its next apply. +- **Two at once.** When two reconciles start the same dead stream at once, say a keepalive tick and a CI apply, the older run is kept and the younger cancels its own start. Of two copies of one config, the older is kept, where 0.33.0 kept the newer. + +On EMR on EC2, a stream whose last step died is started again with `--fluid-restart-of` naming it. This is the end of the new step's arguments, and the line `fluid apply` prints: + +```text +--fluid-checkpoint 1a04314328989295 --fluid-job-sha256 5316fc870295d2ff984b6d407947d73ec4d316fdab2f0f48b3f34ba483d1d9dd --fluid-restart-of s-1 +``` + +```text +Spark ingest-orders (emr-ec2): restarted run s-2 RUNNING on EMR cluster j-1ABCDEF. run s-1 ended FAILED: Driver exited with code 1; started again on the same checkpoint (restart 1 of 5 allowed per rolling hour) +``` + +With five restarts of the last hour already, it starts nothing, exits `1`, and says when the next restart is allowed: + +```text +Spark ingest-orders (emr-ec2): failed run s-r4 FAILED on EMR cluster j-1ABCDEF. run s-r4 ended FAILED: Driver exited with code 1. Not started again: its budget of 5 restarts per rolling hour (execution.retries.maxAttempts, 5 when unset) is spent: s-r0 at 2026-11-19T07:06:11Z, s-r1 at 2026-11-19T07:14:11Z, s-r2 at 2026-11-19T07:22:11Z, s-r3 at 2026-11-19T07:30:11Z, s-r4 at 2026-11-19T07:38:11Z. It can restart from 2026-11-19T08:06:11Z, when fewer than 5 of them are under an hour old; to start it sooner, fix the cause and change the build, and fluid apply starts the new config. Earlier: run s-r3 ended FAILED: Driver exited with code 1; run s-r2 ended FAILED: Driver exited with code 1; run s-r1 ended FAILED: Driver exited with code 1 +``` + +A cluster that ended takes no new step, so the stream cannot start there again. The error names the cluster's state and reason: + +```text +Spark ingest-orders (emr-ec2): failed. EMR cluster j-1ABCDEF is TERMINATED_WITH_ERRORS (INTERNAL_ERROR: The master instance was lost) and takes no new step, so the stream cannot start there again. Point parameters.spark.clusterId at a running cluster and run fluid apply: the stream resumes from its checkpoint +``` + +On EMR Serverless a stream gets a retry policy of `execution.retries.maxAttempts` attempts per hour (5 when unset), so a run that ended `FAILED` has spent them. This is what `fluid apply` says then, for the default of 5: + +```text +run jr-1 ended FAILED: Job failed: OutOfMemoryError. EMR Serverless retries a failed stream (retryPolicy.maxFailedAttemptsPerHour) from execution.retries (5), and it stopped retrying: forge does not start it again on top of the service's own retries, which would fail the same way; fix the cause and change the build, and fluid apply starts the new config +``` + +On EMR on EKS the service retries only as often as `execution.retries` says. A build that sets none is not retried, and the error says so: + +```text +run jr-1 ended FAILED: Job failed: OutOfMemoryError. EMR on EKS retries a failed driver (retryPolicyConfiguration.maxAttempts) only as often as execution.retries says, and this build sets none, so it was not retried: forge does not start a failed run there again. Set execution.retries, or fix the cause and change the build, and fluid apply starts the new config +``` + +On a local build the same rules hold on this machine. When the cause was the machine's, say `spark-submit` or its properties file, the budget error names the run directory to remove: removing `.fluid//runtime/spark//runs` resets the build's history, and with it the budget. This is a plain `fluid apply` on a stream whose job fails at once, on 0.33.0's code and on `main` (the paths are shortened to ``): + +```text +Spark ingest-orders (local): failed run 20261119T075156Z-3449d58d FAILED on this machine (.fluid/commerce.orders-stream/runtime/spark/ingest-orders). run 20261119T075156Z-3449d58d ended FAILED (exit 1): log /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs/20261119T075156Z-3449d58d/driver.log: the stream failed. It is not started again with the same config, which would fail the same way: fix the cause and change the build, and fluid apply starts the new config; when the cause was this machine's (spark-submit, its properties file), remove /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs/20261119T075156Z-3449d58d and fluid apply starts it again +``` + +```text +Spark ingest-orders (local): restarted run 20261119T075203Z-c2b31396 FAILED on this machine (.fluid/commerce.orders-stream/runtime/spark/ingest-orders). run 20261119T075200Z-104292e4 ended FAILED (exit 1): log /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs/20261119T075200Z-104292e4/driver.log: the stream failed; started again on the same checkpoint (restart 1 of 5 allowed per rolling hour); run 20261119T075203Z-c2b31396 ended FAILED (exit 1): log /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs/20261119T075203Z-c2b31396/driver.log: the stream failed +``` + +The second run ended at once as well, so that apply exits `1`: a restart is not a recovery, and the next apply restarts it again until the budget is spent. + +The outputs are from forge-cli's reconcile step. The EMR on EC2 ones are the code `fluid apply` runs after `tofu apply`, with AWS answered by botocore's test stubs: no cluster was touched. The EMR Serverless and EMR on EKS ones are the texts of the same code for a run that ended `FAILED`. The local ones are the real `fluid apply` with a stand-in `spark-submit`. + ### An EMR on EC2 step starts once its policy is attached *(since 0.33.0, [forge-cli #1005](https://github.com/Agenticstiger/forge-cli/pull/1005))* -A step on an `emr-ec2` cluster runs as the role of the cluster's instance profile, which the cluster's owner holds. Forge does not change that role: it belongs to the cluster's owner. `fluid apply` creates the build's least-privilege policy (the managed `aws_iam_policy` in [What forge provisions](#what-forge-provisions)), and the owner attaches it. On 0.32.0 the first `fluid apply` started the stream before anyone could attach the policy, so, unless the role already held the job's permissions, the step failed at once. A stream that failed is not started again with the same config. +A step on an `emr-ec2` cluster runs as the role of the cluster's instance profile, which the cluster's owner holds. Forge does not change that role: it belongs to the cluster's owner. `fluid apply` creates the build's least-privilege policy (the managed `aws_iam_policy` in [What forge provisions](#what-forge-provisions)), and the owner attaches it. On 0.32.0 the first `fluid apply` started the stream before anyone could attach the policy, so, unless the role already held the job's permissions, the step failed at once. In 0.33.0 a stream that failed is not started again with the same config. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* It is started again on its checkpoint, within the [restart budget](#the-restart-budget). Before it starts a step, `fluid apply` now reads the cluster's instance profile and the policies attached to the role in it, and starts nothing until the build's policy is among them. The refusal is the build's error and `fluid apply` ends with exit code `1`. The infrastructure is applied by then, so the policy exists, and the line gives the command that attaches it: @@ -1103,6 +1168,8 @@ Scope `iam:PassRole` to the forge roles, so that a contract cannot pass a strong *On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* A start on AWS also reads the build's job config and runner from S3, so the role needs `s3:GetObject` on both objects: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). The generated `fluid--spark-caller` policy holds it in an `Artifacts` statement, and holds no `kms:Decrypt`. +*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* The policy also covers the product's always-on streams: it lets a role start, watch and stop their runs and read their job config and runner, so that a keepalive can run [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode) from the scheduler's role (the MWAA role, say). It has one statement for each runtime, named `Run`, that lists the runtime's actions once for every target of the product's builds on it, one `PassRole` statement for each runtime that passes a role (EMR Serverless and EMR on EKS), and one `Artifacts` statement. Two more appear where they apply: `StepRoleEmrEc2`, the two IAM reads of an EMR on EC2 start, and `FindApplication`, `emr-serverless:ListApplications` on `*`, for an EMR Serverless build that names no `applicationId`. A managed policy holds at most 6,144 characters, and a statement for each build reached that limit at about six builds. Each build's target, role and artifacts are still named in them. A bucket that encrypts the artifacts with a KMS key also needs `kms:Decrypt` on its key for the scheduler's role, which forge does not grant. `--mode reconcile` reads the uploaded job config on every run, so a role that lacks `s3:GetObject` on it fails while the stream is healthy. + ### When a build leaves the contract *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* When a plan deletes the resources of a Spark build on AWS, because the build left the contract or the product went, `fluid apply` stops the build's runs before `tofu apply` destroys the role, the artifacts and the log group that they run with. Before this change the apply had no step for this. forge-cli's changelog says that it destroyed those and left the stream going, a writer that nothing reconciles any more, and that a destroy of a whole product failed part way, because EMR Serverless deletes only an application that is stopped or created and IAM deletes no policy that is still attached. @@ -1139,7 +1206,7 @@ After the gate, `fluid apply` reads the plan's deletes, and before `tofu apply` Spark ingest-orders (emr-ec2): stopped on EMR cluster j-1ABCDEF. cancelled s-1 before the apply destroys what it runs with ``` -3. **It empties and stops the product's own EMR Serverless application, when that goes.** Every active run in it is cancelled, whichever build started it, and then the application is stopped and waited for, unless it is already stopped or created. An application that the contract only names is not one the plan deletes, so it is left as it is. +3. **It empties and stops the product's own EMR Serverless application, when that goes.** Every active run in it is cancelled, whichever build started it, and then the application is stopped and waited for, unless it is already stopped or created. *On forge-cli `main`, not yet in a release ([forge-cli #1017](https://github.com/Agenticstiger/forge-cli/pull/1017)).* An application that is still `CREATING` or `STARTING` is waited for until it settles first: one that has been created is deleted as it is, and one that has started is stopped. One still moving after the wait refuses the apply. An application that the contract only names is not one the plan deletes, so it is left as it is. ```text EMR Serverless application fluid-commerce.orders-stream-spark (00fabc123def): cancelled jr1 before the apply destroys what it runs with; stopped the application. @@ -1152,7 +1219,7 @@ A run that is still going after the wait, an application that has not stopped af ```text Spark ingest-orders (emr-ec2): run s-1 was cancelled but is still CANCEL_PENDING after 3 s (FLUID_SPARK_WAIT_SECONDS). The plan destroys what these runs run with, so nothing was destroyed. Run fluid apply again once the runs have stopped, or stop them yourself. -The identity running fluid apply needs the runtime's list and cancel actions (as for starting runs). +The identity running fluid apply needs the runtime's list and cancel actions (as for starting runs), iam:ListEntitiesForPolicy for a policy it destroys, and emr-serverless:GetApplication and emr-serverless:StopApplication for the product's own EMR Serverless application. ``` With `--dry-run`, the apply reads the policies and the runs, changes nothing, and says what it would do. A policy that is still attached refuses a dry run too, with the same error and exit code 1. An AWS error while it reads the runs is then a note and not a refusal: @@ -1183,7 +1250,7 @@ What forge does with the cluster decides what the layout must protect: - **The driver is the step's own process.** forge starts the stream with `spark-submit --deploy-mode client`, so that cancelling the step stops the job. AWS's Spark step guide says client mode launches the driver on the cluster's primary instance. - **The checkpoint is in S3, not on the cluster.** It defaults to `s3:///_checkpoints///`, so losing nodes loses no offsets. -- **A step that ends `FAILED` or `INTERRUPTED` counts as failed, and `fluid apply` does not start it again with the same config.** It names the run and its state and starts nothing, because the same config would fail the same way (see [Starting and reconciling runs](#starting-and-reconciling-runs)). To bring the stream back, change the build. After a lost cluster, launch a replacement with forge's policy attached to its instance-profile role, and set `clusterId` to it, with its `emr-ec2:` entry in `FLUID_SPARK_TARGET_ALLOWLIST`: the target is part of the config hash, so `fluid apply` starts the stream on the same checkpoint. +- **A step that ends `FAILED` or `INTERRUPTED` counts as failed, and `fluid apply` does not start it again with the same config.** It names the run and its state and starts nothing, because the same config would fail the same way (see [Starting and reconciling runs](#starting-and-reconciling-runs)). To bring the stream back, change the build. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* `fluid apply` and `--mode reconcile` start such a step again on the stream's checkpoint, within the [restart budget](#the-restart-budget): an `INTERRUPTED` step, which died with its node, is a restart like any other, and the error is for a spent budget. After a lost cluster, launch a replacement with forge's policy attached to its instance-profile role, and set `clusterId` to it, with its `emr-ec2:` entry in `FLUID_SPARK_TARGET_ALLOWLIST`: the target is part of the config hash, so `fluid apply` starts the stream on the same checkpoint. The checklist: diff --git a/docs/cli/apply.md b/docs/cli/apply.md index b28360d..37f8790 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -136,6 +136,7 @@ Two things differ between the engines, and the rest of this page says which one | `amend-and-build` | Same as `amend`, then run the contract's `builds[]` (the build runner). | | `replace` | Destructive. Requires `--allow-data-loss`. | | `replace-and-build` | Same as `replace`, then the builds with a full refresh. Destructive. | +| `reconcile` | *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* Deploys nothing: keeps the contract's always-on Spark streams running. See [Reconcile mode](#reconcile-mode). | What each engine does with the mode: @@ -152,12 +153,75 @@ The build-augmented modes (`amend-and-build`, `replace-and-build`) run the confi amend-and-build (or replace-and-build) to run that build, or drop --build-id. ``` -Before 0.16.3 the flag was accepted and dropped, so a DDL-only apply exited 0. +Before 0.16.3 the flag was accepted and dropped, so a DDL-only apply exited 0. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* `--build-id` is accepted with `--mode reconcile` too, where it names the stream to keep running, and the hint of this refusal adds `--mode reconcile to keep that stream running`. + +*On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* A dbt or Python build runs once under `--mode amend-and-build` whatever its `execution.trigger.type`, `manual` and `continuous` aside: see [What the DAG does](../advanced/airflow.md#what-the-dag-does). Which dbt a dbt build runs on, and what the container fallback installs, are in [Which dbt `fluid apply` runs](./generate.md#which-dbt-fluid-apply-runs). *(since 0.30.0, [forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861))* On Databricks, a dbt build with `execution.runtime.platform: databricks` runs as the job the apply provisions. The build phase starts a run of the job and waits for it to end, and a failed run fails the apply. See [Running the job from `fluid apply`](../providers/databricks.md#running-the-job-from-fluid-apply). *(since 0.30.0, [forge-cli #872](https://github.com/Agenticstiger/forge-cli/pull/872))* A build with `engine: lakeflow` runs as the Lakeflow pipeline the apply provisions. The build phase starts an update of the pipeline and waits for it to end, and a failed update fails the apply. A continuous pipeline is not started: only its table's tags are set. See [Running it from `fluid apply`](../providers/databricks.md#running-it-from-fluid-apply). +### Reconcile mode + +*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* `--mode reconcile` keeps the contract's always-on Spark streams running, and deploys nothing: it runs no `tofu`, no DDL and no build, and it takes no plan. A stream that died is started again on its checkpoint, within its [restart budget](../advanced/spark-iceberg-aws.md#the-restart-budget), when what the last `fluid apply` deployed is what the contract compiles to. A scheduler task that runs it every few minutes, a keepalive, keeps a stream going between applies: + +```bash +fluid apply contract.fluid.yaml --env prod --mode reconcile --build-id ingest-orders --yes +``` + +- **What it reads.** The contract, which goes through the same schema gate, `--build-id` check and apply-time hooks as any apply. A `plan.json` is refused with `apply_reconcile_needs_contract`. On AWS it also reads the job config that the last apply uploaded, from S3, on every run, so a missing `s3:GetObject` shows while the stream is healthy. It reads the runs from the runtime's own history, and keeps no state of its own. +- **Which streams.** With `--build-id`, that one stream. A build that is not a forge Spark build, a batch, or a stream triggered `availableNow` is refused with `apply_reconcile_not_a_stream`: its scheduler runs it with `--mode amend-and-build --build-id `. Without `--build-id`, every always-on Spark stream of the contract. A contract with none prints `No always-on Spark stream in the contract: nothing to reconcile.` and exits `0`. +- **What it never does.** It does not replace a config, and it does not stop a build the contract no longer holds: a full `fluid apply` does both. It cancels only a younger second copy of the running config. +- **`--dry-run`** reads the same state and starts nothing. On a terminal, without `--yes`, it asks before it starts or cancels a run. An empty or a closed answer is a no: it starts or cancels nothing, says so (`declined`), and exits `0`. +- **The settings of the apply** reach it too: the operator's lists (`FLUID_SPARK_TARGET_ALLOWLIST`, which also lists a dead-letter topic, `FLUID_SPARK_CONTRACT_SQL_ALLOWLIST`, `FLUID_SPARK_LOCAL_NAMESPACE_ALLOWLIST` for a `runtime: local` stream, and the host allowlists) gate every reconcile, and a refused build is an error, `apply_spark_refused`. A keepalive's environment needs the ones the CI apply has. [`FLUID_SPARK_WAIT_SECONDS`](../advanced/environment-variables.md#spark-on-aws-preview), 900 seconds by default, bounds how long it waits for a started run to leave its pending states and for a cancelled one to stop. + +For each stream it finds one of these, and exits `1` when any stream is an error: + +| It finds | It does | Shown as | Exit | +|---|---|---|---| +| A run of the deployed config going | Nothing. | `unchanged` | `0` | +| The last run died: it ended `FAILED` or `INTERRUPTED`, was lost with its machine, node or cluster, was stopped by a signal from outside forge, or the stream stopped by itself | Starts it again on the same checkpoint, within the budget. | `restarted` | `0`, or `1` when the new run ends at once | +| The budget is spent | Starts nothing. It names the budget, the restarts and when the next one is allowed. | `failed` | `1` | +| The last run was cancelled through the runtime, or is being stopped | Leaves it stopped. A plain `fluid apply` starts a cancelled stream again. | `left-stopped` | `0` | +| What is deployed is not the contract: the uploaded job config differs, the last run ran another config, another config is running, nothing is deployed, or no run is recorded | Starts nothing, and says to run `fluid apply`. | `refused` | `1` | +| A younger second copy of the running config | Cancels it, and keeps the older run. | `unchanged` | `0` | +| The last run ended `FAILED` on EMR Serverless or EMR on EKS | Starts nothing: the service's own retries are spent. | `failed` | `1` | +| An EMR on EC2 cluster that ended | Starts nothing, and names the cluster's state and reason. | `failed` | `1` | +| A topic or a cluster other than the one that the stream's checkpoint was written for | Starts nothing: a checkpoint holds the offsets of one topic on one cluster. See [Starting and reconciling runs](../advanced/spark-iceberg-aws.md#starting-and-reconciling-runs). | `refused` | `1` | +| An AWS error while it reads the runs or starts one | Says what failed, and what it had done before it. | `failed` | `1` | + +A stream of a local build on this machine, killed from outside, then reconciled, from a real run (the paths are shortened to ``): + +```text +Spark ingest-orders (local): unchanged run 20261119T075039Z-181dd129 RUNNING on this machine (.fluid/commerce.orders-stream/runtime/spark/ingest-orders) +``` + +```text +Spark ingest-orders (local): restarted run 20261119T075054Z-157881b9 RUNNING on this machine (.fluid/commerce.orders-stream/runtime/spark/ingest-orders). run 20261119T075039Z-181dd129 ended FAILED (exit -9): log /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs/20261119T075039Z-181dd129/driver.log: stand-in spark-submit; started again on the same checkpoint (restart 1 of 5 allowed per rolling hour) +``` + +With the contract changed since the last apply, it starts nothing and exits `1`: + +```text +Spark ingest-orders (local): refused on this machine (.fluid/commerce.orders-stream/runtime/spark/ingest-orders). the last run 20261119T075106Z-5dd7554c ran config b2daa0daab82, and the contract compiles to f79cf67ae81e. --mode reconcile changes no config: run fluid apply to start the new config +``` + +`--dry-run` on a stream that died: + +```text +Spark ingest-orders (local): planned on this machine (.fluid/commerce.orders-stream/runtime/spark/ingest-orders). would start it again on the same checkpoint: run 20261119T075129Z-cc92dd21 ended FAILED (exit -9): log /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs/20261119T075129Z-cc92dd21/driver.log: stand-in spark-submit (restart 1 of 5 allowed per rolling hour) +``` + +A plan is not a contract: + +```text +❌ apply_reconcile_needs_contract [ERR_APPLY_RECONCILE_NEEDS_CONTRACT] + path: /plan.json + hint: --mode reconcile applies no plan: it reads the contract, what the last fluid apply deployed and how the runs are doing. Pass the contract (contract.fluid.yaml). +``` + +These outputs are from the real `fluid apply` on `main`, on a stream of `runtime: local` with a stand-in `spark-submit` that sleeps, killed with `kill -9`: no Spark, no cloud and no AWS call. The runs on AWS go through the same code with the runtime's own history: [the restart budget](../advanced/spark-iceberg-aws.md#the-restart-budget) shows them with AWS answered by botocore's test stubs. + ## Contract validation *([forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714), since 0.20.0)* `fluid apply` checks the contract against its schema before it plans anything, on both engines and in every mode, `--dry-run` and the build modes included. A GCP contract that still carries the deprecated `metadata.policies` mapping stops before `tofu` runs: @@ -720,7 +784,7 @@ Options marked *native* are read only by the native engine. As of 0.18.1 the Ope | Option | Default | Description | | --- | --- | --- | -| `--mode` | `amend` | One of `dry-run`, `create-only`, `amend`, `amend-and-build`, `replace`, `replace-and-build`. | +| `--mode` | `amend` | One of `dry-run`, `create-only`, `amend`, `amend-and-build`, `replace`, `replace-and-build`. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* Also `reconcile`: see [Reconcile mode](#reconcile-mode). | | `--yes` | off | Skip confirmation. | | `--dry-run` | off | Alias for `--mode dry-run`. | | `--ensure-opentofu` | off | *(since 0.8.8)* If the `tofu` binary is missing, provision a **pinned, SHA-256-verified** OpenTofu build before a cloud apply — **no root, gpg, cosign, curl, or unzip needed** (Python stdlib only). Idempotent (a usable `tofu` at/above the engine floor is left untouched) and a no-op for native / `local` applies. Pin via `FLUID_OPENTOFU_VERSION`. `fluid generate ci` bakes this into the generated apply stage so cloud applies work on locked-down / non-root runners. | diff --git a/docs/upgrading.md b/docs/upgrading.md index 60f8626..503ccc6 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1484,6 +1484,16 @@ These changes are on forge-cli `main`. The pin on this site is 0.33.0, so they a - **A Spark job on AWS can write its own table where Lake Formation governs new tables.** forge granted a job's role Lake Formation access to its table only when the output's binding declared a non-empty `governance.lakeFormation` block, so in an account that removed the `IAM_ALLOWED_PRINCIPALS` default for new tables, a stream without that block could not see its own table. The grant (`INSERT`, `ALTER`, `DESCRIBE`, `SELECT`, and `DELETE` for a batch job) is now emitted for such an output and for a stream's dead-letter table beside it, with a `count` that the plan decides from the account's settings: 1 where Lake Formation governs new tables, 0 where IAM alone decides, so an account that keeps the default creates nothing new. The identity that runs `fluid plan` and `fluid apply` needs `lakeformation:GetDataLakeSettings`. An output with a non-empty `governance.lakeFormation` block keeps its grants, and an EMR on EC2 job, which runs as the cluster's own role, gets none. See [Lake Formation writes](./advanced/spark-iceberg-aws.md#lake-formation-writes). +*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* + +- **`fluid apply --mode reconcile` keeps the Spark streams running and deploys nothing.** It runs no `tofu`, no DDL and no build, and it takes no plan. It starts a stream that died again on its checkpoint, within its restart budget, when what the last apply deployed is what the contract compiles to. When it is not, it starts nothing, exits `1` and names what to run. `--build-id ` picks one stream, and a keepalive, a scheduler task every few minutes, is what runs it. See [Reconcile mode](./cli/apply.md#reconcile-mode). +- **A Spark stream that died starts again within a restart budget, on a plain `fluid apply` too.** On EMR on EC2, Glue and a local build, a stream whose last run ended `FAILED` was an error until the build changed. It is started again on its checkpoint now, up to `execution.retries.maxAttempts` times per rolling hour (5 when unset, at most 10), each restart carrying `--fluid-restart-of `, and a spent budget is the error. EMR Serverless and EMR on EKS keep their own retries, and forge starts nothing on top of them. A run cancelled through the runtime is left stopped by `--mode reconcile` and started again by a plain `fluid apply`. `execution.retries` is no longer part of the config hash on EMR on EC2 and Glue, so a stream there that declares it restarts once on its next apply. A local run whose supervisor got a signal from outside forge, a shutdown of the machine say, reads as failed, not cancelled. Of two copies of one config the older is kept, where the newer was. See [The restart budget](./advanced/spark-iceberg-aws.md#the-restart-budget). +- **The scheduler policy `fluid--spark-caller` covers always-on streams, and its statements change shape.** It lets a role start, watch and stop a stream's runs and read its job config and runner, so a keepalive can run on MWAA. It holds one statement for each runtime, with every build's target, role and artifacts, where it held one for each build, which reached the 6,144-character limit of a managed policy at about six builds. It holds no `kms:Decrypt`: a bucket that encrypts the artifacts with a KMS key needs it for the scheduler's role. See [The caller's permissions](./advanced/spark-iceberg-aws.md#the-caller-s-permissions). + +*On forge-cli `main`, not yet in a release ([forge-cli #1017](https://github.com/Agenticstiger/forge-cli/pull/1017)).* + +- **A destroy waits for an EMR Serverless application that is still being created or started.** A `CREATING` or `STARTING` application is waited for until it settles: a created one is deleted as it is, a started one is stopped first, and one still moving after `FLUID_SPARK_WAIT_SECONDS` refuses the apply. The refusal's hint names `iam:ListEntitiesForPolicy`, `emr-serverless:GetApplication` and `emr-serverless:StopApplication`, which the identity that runs the destroy needs beyond the runtime's list and cancel actions. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract). + ## Older releases Upgrading from before `0.15.0`? Start with the