Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/advanced/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ fluid apply runtime/plan.json --bundle runtime/bundle.tgz --env aws --yes
| `FLUID_SPARK_EXECUTION_ROLE_ALLOWLIST` | The roles `parameters.spark.executionRoleArn` may name: a role ARN, or an ARN prefix ending in `/`. |
| `FLUID_SPARK_CONF_ALLOWLIST` | `sparkConf` keys beyond the built-in tuning keys: a key, or a prefix ending in `*`. A key that loads code, sets credentials or moves the catalog or checkpoint is refused even when listed. |
| `FLUID_SPARK_CONTRACT_SQL_ALLOWLIST` | The builds whose own `properties.sql` a Spark job may run: `<product>`, `<product>/<build>`, or `<product>/<build>@sha256:<digest>`. *(since 0.33.0, [forge-cli #990](https://github.com/Agenticstiger/forge-cli/pull/990))* `fluid validate` reads it too, and warns only about a build it does not approve: see [Contract SQL and `fluid validate`](./spark-iceberg-aws.md#contract-sql-and-fluid-validate). |
| `FLUID_SPARK_WAIT_SECONDS` | How long a reconcile waits for a cancelled run to stop and a started one to leave its pending states. Default 900. |
| `FLUID_SPARK_WAIT_SECONDS` | How long a reconcile waits for a cancelled run to stop and a started one to leave its pending states. Default 900. *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* It is also how long `fluid apply` waits, for each run, for the runs and the application that a destroy would leave going: see [When a build leaves the contract](./spark-iceberg-aws.md#when-a-build-leaves-the-contract). A value that is not a positive number is ignored. |
| `FLUID_SPARK_MAVEN_REPOSITORY` | One https Maven mirror that replaces Confluent's repository in `spark.jars.repositories`. A URL with a user or password, a query, a comma, plain http or a link-local host is refused. |
| `FLUID_SPARK_GLUE_JARS_URL` | `s3://<bucket>/<prefix>/`: where the platform keeps the pinned jars of a Glue Confluent Avro build. |
| `FLUID_SPARK_SUBMIT_EXECUTABLE` | The `spark-submit` a build of `parameters.spark.runtime: local` starts: an absolute path to an executable outside the project, or a name on `PATH`. Default: `spark-submit` on `PATH`. *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* [Running a build on this machine](./spark-iceberg-aws.md#operator-settings-for-a-local-run). `fluid doctor --env` lists it. |
Expand Down
58 changes: 58 additions & 0 deletions docs/advanced/spark-iceberg-aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -1016,6 +1016,62 @@ The role that runs `fluid apply`, or the scheduler's execution role (the MWAA ro

Scope `iam:PassRole` to the forge roles, so that a contract cannot pass a stronger one.

### When a build leaves the contract

*On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* When a plan deletes the resources of a Spark build on AWS, because the build left the contract or the product went, `fluid apply` stops the build's runs before `tofu apply` destroys the role, the artifacts and the log group that they run with. Before this change the apply had no step for this. forge-cli's changelog says that it destroyed those and left the stream going, a writer that nothing reconciles any more, and that a destroy of a whole product failed part way, because EMR Serverless deletes only an application that is stopped or created and IAM deletes no policy that is still attached.

`fluid apply` reads the plan's deletes, and before `tofu apply` it does these:

1. **It refuses when a policy it would destroy is still attached.** forge emits two IAM managed policies that someone else attaches: an EMR on EC2 build's job policy, which the cluster's owner attaches to the cluster's instance-profile role, and the product's scheduler policy `fluid-<product>-spark-caller`, which the platform attaches to the scheduler's role. The apply ends with the error `opentofu_destroy_preparation_failed` and the kind `spark-policy-attached`. Nothing is stopped or destroyed. The error names where the policy is attached and the commands that detach it. Its text and remediation read:

```text
IAM policy fluid-commerce.orders-stream-ingest-orders-744ba50a-spark (an EMR on EC2 build's job policy, which the cluster's owner attaches to its instance-profile role) is attached to role EMR_EC2_DefaultRole. The plan destroys it, and IAM deletes no attached policy (DeleteConflict), so the apply would stop part way through the destroys. Nothing was stopped or destroyed.
Detach it, then run fluid apply again:
aws iam detach-role-policy --role-name EMR_EC2_DefaultRole --policy-arn arn:aws:iam::000000000005:policy/fluid-commerce.orders-stream-ingest-orders-744ba50a-spark
```

2. **It stops the runs of each build that goes.** A build's runs are looked for where the tags of its log group say they are (see [the tags](#the-tags-that-say-where-a-build-runs)), or, for an EMR on EC2 build, of its job policy. A Glue job that goes has its runs stopped too, even when the build stays: a build moved off Glue keeps its log group. Every active run that the runtime's list returns is cancelled, and one that is already cancelling is only waited for. The apply waits for each to end. EMR Serverless, EMR on EC2 and EMR on EKS runs are found by the build's run name, and a Glue job's runs by the job. Glue lists the 25 most recent runs of a job, and EMR on EKS one page of them, so stop an older run yourself. The builds are taken in the plan's order.

```text
Spark ingest-orders (emr-ec2): stopped on EMR cluster j-1ABCDEF. cancelled s-1 before the apply destroys what it runs with
```

3. **It empties and stops the product's own EMR Serverless application, when that goes.** Every active run in it is cancelled, whichever build started it, and then the application is stopped and waited for, unless it is already stopped or created. An application that the contract only names is not one the plan deletes, so it is left as it is.

```text
EMR Serverless application fluid-commerce.orders-stream-spark (00fabc123def): cancelled jr1 before the apply destroys what it runs with; stopped the application.
```

A build that moves from an EMR on EC2 cluster to another runtime keeps its log group, whose tags then name the new runtime. Its job policy goes, and carries the old cluster, so the steps on the old cluster are cancelled as well. A build that moves from EMR Serverless or EMR on EKS to another runtime keeps its log group too, and the apply does not look for its old runs then: stop them yourself.

A run that is still going after the wait, an application that has not stopped after it, or an AWS error on a run or on the application, refuses the apply before anything is destroyed, with the error `opentofu_destroy_preparation_failed` and the kind `spark-destroy-runs-going`. An AWS answer that the target no longer exists is not an error: nothing runs there. The wait is [`FLUID_SPARK_WAIT_SECONDS`](./environment-variables.md#spark-on-aws-preview) for each run and for the application's stop, 900 seconds by default. Run `fluid apply` again once the runs have ended, or stop them yourself. The text and remediation of the error read:

```text
Spark ingest-orders (emr-ec2): run s-1 was cancelled but is still CANCEL_PENDING after 3 s (FLUID_SPARK_WAIT_SECONDS). The plan destroys what these runs run with, so nothing was destroyed.
Run fluid apply again once the runs have stopped, or stop them yourself.
The identity running fluid apply needs the runtime's list and cancel actions (as for starting runs).
```

With `--dry-run`, the apply reads the policies and the runs, changes nothing, and says what it would do. A policy that is still attached refuses a dry run too, with the same error and exit code 1. An AWS error while it reads the runs is then a note and not a refusal:

```text
Spark ingest-orders (emr-ec2): planned on EMR cluster j-1ABCDEF. would cancel s-1
```

#### The tags that say where a build runs

The log group of every Spark build, and an EMR on EC2 build's job policy, carry three tags: `fluid_runtime`, `fluid_run_target` (`applicationId=<id>`, `glueJobName=<name>`, `clusterId=<id>` or `virtualClusterId=<id>`) and `fluid_run_region`. A destroy reads them from the state, because the build is no longer in the contract. The first `fluid apply` after the upgrade adds them to the existing resources in place. A build that a forge without them deployed, and that goes in that same apply, has none, and the apply cannot tell from them where its runs are. It prints a note, and stops no run of that build, unless the plan also deletes the build's Glue job or the product's own EMR Serverless application: the runs of a Glue job, and every run in that application, are stopped without the tags.

```text
note: Spark build ingest-orders goes, but nothing it is destroyed with says where it runs (an apply by an earlier forge): stop its run fluid-commerce.orders-stream-ingest-orders-744ba50a yourself if one is going.
```

#### What the identity needs

Besides the list, get and cancel actions of the runtime that the removed build ran on, which can be a runtime the contract no longer names (see [The caller's permissions](#the-caller-s-permissions)), the identity that runs `fluid apply` needs `iam:ListEntitiesForPolicy`, to tell whether a policy is attached, and, for the product's own EMR Serverless application, `emr-serverless:ListJobRuns`, `GetJobRun` and `CancelJobRun` as well as `emr-serverless:GetApplication` and `emr-serverless:StopApplication`.

The outputs above come from forge-cli's destroy step, the code `fluid apply` calls with the plan's deletes before `tofu apply`, with AWS answered by botocore's test stubs and the plan's deletes built as forge-cli's own tests build them: no AWS account, cluster or application was touched, and the printed account number is a placeholder.

## Running EMR on EC2 on Spot

A stream on EMR on EC2 runs as one step of a cluster you already run (`clusterId`). forge creates no cluster, so the layout below is a checklist for the cluster's owner. Only the `sparkConf` values in items 4 and 6 go in the contract. Its aim is that a Spot interruption costs time and not records.
Expand Down Expand Up @@ -1183,6 +1239,8 @@ In the account's `<account>-fluid-data` fallback bucket, and in a `shared` packa
| `spark-emr-application-shared` | An EMR Serverless build with no `applicationId` and no `emrApplication: isolated` |
| `spark-checkpoint-scope` | A declared checkpoint outside `s3://<bucket>/_checkpoints/<product>/`, while forge creates the role |
| `spark-glue-jars-unset`, `spark-glue-jars-url` | A Glue Avro build, and `FLUID_SPARK_GLUE_JARS_URL` is unset or malformed |
| `spark-policy-attached` | *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* The plan destroys an IAM policy that is still attached: an EMR on EC2 build's job policy or the product's scheduler policy. The error is `opentofu_destroy_preparation_failed`. See [When a build leaves the contract](#when-a-build-leaves-the-contract). |
| `spark-destroy-runs-going` | *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* A run of a build that goes, or the product's own application, is still going after `FLUID_SPARK_WAIT_SECONDS`, or AWS refused to list, cancel or stop it. The error is `opentofu_destroy_preparation_failed`. See [When a build leaves the contract](#when-a-build-leaves-the-contract). |
| `spark-dead-letters-table-unsupported` | *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* A `deadLetter` table sink beside an output at its bucket's root, beside a location forge registers with Lake Formation, or beside an output whose lifecycle expires its objects. See [What is refused](#what-is-refused). |
| `apply_spark_refused` | `fluid apply` refuses a Spark build before `tofu apply` changes anything when an operator's list does not allow what the build names: an MSK cluster that `FLUID_SPARK_TARGET_ALLOWLIST` does not list, a host, a secret reference, or a `sparkConf` key. *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* It also refuses a `deadLetter` topic that `FLUID_SPARK_TARGET_ALLOWLIST` does not list (see [Listing the topic](#listing-the-topic)), and, for a local build, a namespace that `FLUID_SPARK_LOCAL_NAMESPACE_ALLOWLIST` does not list (see [The table the job creates](#the-table-the-job-creates)). |
| `apply_spark_artifact_mismatch` | The bound plan lists other Spark runs than apply compiles |
Expand Down
8 changes: 8 additions & 0 deletions docs/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -1451,6 +1451,14 @@ These changes are in 0.33.0. The pin on this site stays at 0.32.0 until the page

- **New, opt-in: `fluid ship --journal-dir` keeps a stage journal, and every `--release-dir` run keeps one.** Each stage writes a `started` event before it runs and a `finished` event after, with its status and the digests of what it wrote. Before an `apply` that is not a dry run, an enforcing `policy-apply`, `publish` or a `schedule-sync` that is not a dry run, ship refuses the stage when an earlier attempt of it, or of `apply`, has no known outcome, until you inspect the target and pass `--acknowledge-inspected STAGE`. An `apply` that did not pass is refused until stage 6 has planned again. `--explain` shows what the journal would refuse. Without either flag no journal is kept and no stage is refused, and the run report records a stage killed by a signal (exit 129 to 159) as `unknown` instead of `failed`. See [The stage journal](./cli/ship.md#the-stage-journal).

## Not in a release yet

These changes are on forge-cli `main`. The pin on this site stays at 0.32.0 until they ship.

*On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).*

- **`fluid apply` stops a removed Spark build's runs on AWS before it destroys what they run with.** When a plan deletes a build's resources, because the build left the contract or the product went, the apply cancels the build's active runs and waits for them (`FLUID_SPARK_WAIT_SECONDS`, 900 seconds by default), and empties and stops the product's own EMR Serverless application. It refuses first, with the `aws iam detach-*-policy` commands, when an IAM policy that the plan destroys is still attached, and a run that is still going after the wait refuses the apply before anything is destroyed. `--dry-run` says what it would cancel. The first apply after the upgrade adds three tags, `fluid_runtime`, `fluid_run_target` and `fluid_run_region`, in place to each Spark build's log group and to an EMR on EC2 build's job policy. The identity that runs `fluid apply` needs `iam:ListEntitiesForPolicy`, the list, get and cancel actions of the runtime that a removed build ran on, and, for the product's own application, `emr-serverless:GetApplication` and `emr-serverless:StopApplication` besides the EMR Serverless list, get and cancel actions. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract).

## Older releases

Upgrading from before `0.15.0`? Start with the
Expand Down
Loading