Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions docs/advanced/spark-iceberg-aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -1020,7 +1020,23 @@ Scope `iam:PassRole` to the forge roles, so that a contract cannot pass a strong

*On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* When a plan deletes the resources of a Spark build on AWS, because the build left the contract or the product went, `fluid apply` stops the build's runs before `tofu apply` destroys the role, the artifacts and the log group that they run with. Before this change the apply had no step for this. forge-cli's changelog says that it destroyed those and left the stream going, a writer that nothing reconciles any more, and that a destroy of a whole product failed part way, because EMR Serverless deletes only an application that is stopped or created and IAM deletes no policy that is still attached.

`fluid apply` reads the plan's deletes, and before `tofu apply` it does these:
*On forge-cli `main`, not yet in a release ([forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007)).* Removing a build needs no `--allow-data-loss`, unless the build has a dead-letter table. What a Spark build runs with holds no data, so the [data-loss gate](../cli/apply.md#opentofu-data-loss-gate) does not count it: the runner and the job config in the artifact bucket, the copies of the jars that a Glue build loads, the build's log group, its role and the role's policies (for an EMR on EC2 build, its job policy), the Glue job and its connection, the product's own EMR Serverless application, the scheduler policy `fluid-<product>-spark-caller`, the key that encrypts the product's Spark log groups, and the role that a registered Lake Formation location is written through. [What forge provisions](#what-forge-provisions), [Avro on Glue](#avro-on-glue) and [Lake Formation writes](#lake-formation-writes) say which of them a build gets. forge-cli's changelog says that the gate counted all of them as data, so dropping a build from a contract was refused unless the apply was allowed to destroy data, and that allowance let a real data loss in the same plan through.

`fluid plan --tofu`, `fluid apply --dry-run` and `fluid apply` list them as removed and not gated. For the example contract's build on EMR on EC2, the line reads:

```text
5 resource(s) a removed build ran with (its artifacts, role, log group and job; no table data lost; not gated; its logs go with them): aws_s3_object.commerce_orders_stream_ingest_orders_spark_runner, aws_s3_object.commerce_orders_stream_ingest_orders_spark_config, aws_cloudwatch_log_group.commerce_orders_stream_ingest_orders_spark, aws_iam_policy.commerce_orders_stream_ingest_orders_spark, aws_kms_key.commerce_orders_stream_spark_logs
```

What holds data still counts: an output table, a bucket, and a bucket's own key (its address ends in `_kms`). So does the dead-letter table of a stream with `sink: table`: the build adds it, and it goes when the build goes, so removing that build needs the flag. `fluid apply` refuses a plan that removes any of them unless it is given the flag, its `--dry-run` included, and `fluid plan --tofu` warns that it will. The refusal counts those alone. With the output table in the same plan, the five resources of the build are not part of the count:

```text
error: plan destroys 1 resource(s); `tofu` does not snapshot data — re-run with --allow-data-loss to proceed
```

As for every plan, when the plan's per-resource events do not account for all the removals in its summary (an older `tofu`, a truncated stream), the gate counts every removal and the flag is needed.

After the gate, `fluid apply` reads the plan's deletes, and before `tofu apply` it does these:

1. **It refuses when a policy it would destroy is still attached.** forge emits two IAM managed policies that someone else attaches: an EMR on EC2 build's job policy, which the cluster's owner attaches to the cluster's instance-profile role, and the product's scheduler policy `fluid-<product>-spark-caller`, which the platform attaches to the scheduler's role. The apply ends with the error `opentofu_destroy_preparation_failed` and the kind `spark-policy-attached`. Nothing is stopped or destroyed. The error names where the policy is attached and the commands that detach it. Its text and remediation read:

Expand Down Expand Up @@ -1070,7 +1086,7 @@ The log group of every Spark build, and an EMR on EC2 build's job policy, carry

Besides the list, get and cancel actions of the runtime that the removed build ran on, which can be a runtime the contract no longer names (see [The caller's permissions](#the-caller-s-permissions)), the identity that runs `fluid apply` needs `iam:ListEntitiesForPolicy`, to tell whether a policy is attached, and, for the product's own EMR Serverless application, `emr-serverless:ListJobRuns`, `GetJobRun` and `CancelJobRun` as well as `emr-serverless:GetApplication` and `emr-serverless:StopApplication`.

The outputs above come from forge-cli's destroy step, the code `fluid apply` calls with the plan's deletes before `tofu apply`, with AWS answered by botocore's test stubs and the plan's deletes built as forge-cli's own tests build them: no AWS account, cluster or application was touched, and the printed account number is a placeholder.
The outputs above, but the two of the data-loss gate, come from forge-cli's destroy step, the code `fluid apply` calls with the plan's deletes before `tofu apply`, with AWS answered by botocore's test stubs and the plan's deletes built as forge-cli's own tests build them: no AWS account, cluster or application was touched, and the printed account number is a placeholder. The two outputs of the data-loss gate come from forge-cli's plan review, the code that `fluid plan --tofu` and `fluid apply` run on the plan's deletes after `tofu plan`. The deletes given to it are the resources that the example contract's build adds to the module, which a plan deletes when the build is removed, and, for the second output, the contract's output table: no `tofu`, state or AWS call was involved.

## Running EMR on EC2 on Spot

Expand Down
2 changes: 2 additions & 0 deletions docs/cli/apply.md
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,8 @@ Removing a resource that only grants access is a revocation, not a loss, and pas

A Cloud KMS key's IAM grant is not exempt: without it BigQuery cannot decrypt the table. Any removal of a type not in the list counts, so removing a bucket's lifecycle or server-side-encryption configuration, or a product's KMS key, needs `--allow-data-loss` even though no object is deleted. See [AWS](../providers/aws.md). If the plan's per-resource events do not account for every removal in its summary (an older `tofu`, a truncated stream), the gate counts every removal.

*On forge-cli `main`, not yet in a release ([forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007)).* On AWS the list is by type and, for what a forge Spark build ran with, by the end of the resource's name too. Its artifacts, role, log group and the key that encrypts the log groups, jobs and the product's own EMR Serverless application hold no data, so removing a Spark build is not a data loss. `apply` lists them (`<n> resource(s) a removed build ran with (its artifacts, role, log group and job; no table data lost; not gated; its logs go with them)`) and passes without the flag. An output table, a bucket and a bucket's own key still count, and so does a stream's dead-letter table (`sink: table`), which goes with the build: removing that build needs the flag. See [When a build leaves the contract](../advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract).

Some governance changes cannot be made to a live BigQuery table in place and plan its replacement, which this gate refuses without the flag: adding retention (`expire: true`) to a table that exists, adding or changing its Cloud KMS key, and removing `expire`. Roll these out as: plan, review the replacement, apply with `--allow-data-loss`, then run the build again to land the data. See [GCP](../providers/gcp.md).

*(since 0.27.0, [forge-cli #793](https://github.com/Agenticstiger/forge-cli/pull/793))* The gate also holds a plan that destroys nothing but changes existing data in place. On AWS, a Glue table whose column types change plans as a plain update, yet the Parquet files written under the old type may stop reading until they are rewritten. `fluid apply`, its `--dry-run` included, refuses until `--allow-data-loss` is set, and names each table and each `column: old -> new`:
Expand Down
4 changes: 4 additions & 0 deletions docs/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -1459,6 +1459,10 @@ These changes are on forge-cli `main`. The pin on this site stays at 0.32.0 unti

- **`fluid apply` stops a removed Spark build's runs on AWS before it destroys what they run with.** When a plan deletes a build's resources, because the build left the contract or the product went, the apply cancels the build's active runs and waits for them (`FLUID_SPARK_WAIT_SECONDS`, 900 seconds by default), and empties and stops the product's own EMR Serverless application. It refuses first, with the `aws iam detach-*-policy` commands, when an IAM policy that the plan destroys is still attached, and a run that is still going after the wait refuses the apply before anything is destroyed. `--dry-run` says what it would cancel. The first apply after the upgrade adds three tags, `fluid_runtime`, `fluid_run_target` and `fluid_run_region`, in place to each Spark build's log group and to an EMR on EC2 build's job policy. The identity that runs `fluid apply` needs `iam:ListEntitiesForPolicy`, the list, get and cancel actions of the runtime that a removed build ran on, and, for the product's own application, `emr-serverless:GetApplication` and `emr-serverless:StopApplication` besides the EMR Serverless list, get and cancel actions. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract).

*On forge-cli `main`, not yet in a release ([forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007)).*

- **Removing a Spark build on AWS needs no `--allow-data-loss`, unless the build has a dead-letter table.** The data-loss gate counted what a removed build ran with (its runner and job config objects, role and policies, log group and the key its logs are encrypted with, Glue job and connection, and the product's own EMR Serverless application) as data, so a plan that drops a build was refused unless the apply was allowed to destroy data, and that allowance also let a real data loss in the same plan through. They are now listed as removed and not gated. An output table, a bucket and a bucket's own key still count, and so does a stream's dead-letter table (`sink: table`), which goes with the build, so a plan that removes one of them needs the flag as before. A pipeline that passes it only to remove a build that has no dead-letter table no longer needs to. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract).

## Older releases

Upgrading from before `0.15.0`? Start with the
Expand Down
Loading