From bea8f1ad129bdda973b2028e431b09cc1ba51eaa Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 08:38:47 +0100 Subject: [PATCH] docs(spark): a run starts only on the job config fluid apply compiled (forge-cli #1014) The two steps, the three refusals, the digest, and the permissions of the identity that runs fluid apply. --- docs/advanced/spark-iceberg-aws.md | 47 ++++++++++++++++++++++++++++++ docs/upgrading.md | 4 +++ 2 files changed, 51 insertions(+) diff --git a/docs/advanced/spark-iceberg-aws.md b/docs/advanced/spark-iceberg-aws.md index 7d44a63..a33d535 100644 --- a/docs/advanced/spark-iceberg-aws.md +++ b/docs/advanced/spark-iceberg-aws.md @@ -740,6 +740,13 @@ spark.sql.shuffle.partitions=16 1a04314328989295 ``` +*On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* The argv of a run on `main` ends with two more lines, `--fluid-job-sha256` and the digest of the job config that forge wrote under the root, and the runner refuses any other config before it reads anything: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). These are the two lines in a run of the same stream on `main`: + +```text +--fluid-job-sha256 +273573946cb2003dc5cb3d54ecb3223eada561317cd2ed8df658b9c6fc11e22f +``` + forge puts the properties file's path on the command line, and copies none of its values there, except the two lists it merges, below. forge passes the settings of the run beside the file's: the contract's allowed `sparkConf` ([Spark settings](#spark-settings)), the Iceberg extension and the Kafka connector package. For `spark.jars.packages` and `spark.sql.extensions`, which `spark-submit` would otherwise replace as a whole, forge passes the union of the file's value and its own, the file's entries first. A package that both name is passed once, in the file's version. The file's own `spark.jars` and `spark.jars.repositories` replace forge's, so on a Spark that is not an EMR image the file names the Iceberg runtime jar there. Without its own `spark.jars`, forge points `spark.jars` at the path an EMR image keeps it, `/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar`. ### Operator settings for a local run @@ -1003,6 +1010,44 @@ The line is printed unbroken, and a narrow terminal wraps it at spaces: join the The output above is from forge-cli's reconcile step, the code `fluid apply` runs after `tofu apply`, with AWS answered by botocore's test stubs: the real cluster, instance profile and role calls were not made. +### A run starts only on the config this apply compiled + +*On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* A run's job config is an object in S3, which the runtime reads when the run starts and again each time it starts the job again after a failure. Until this change nothing checked what the object held: a config that another `fluid apply` had put there since this one compiled its runs would run under the config hash of this apply's runs, and a later `fluid apply` would take the hash for the config. Now a run starts only on the config that `fluid apply` compiled, in two steps. + +1. **`fluid apply` reads the job config and the runner back from S3 before it starts a run, and starts nothing unless both are the ones it compiled.** This is for the runs on AWS: EMR Serverless, Glue, EMR on EC2 and EMR on EKS. The job config counts when its [digest](#the-digest) is the compiled config's. The runner counts when it is, byte for byte, the runner that this forge-cli ships. On EMR on EC2 the [check that the policy is attached](#an-emr-on-ec2-step-starts-once-its-policy-is-attached) comes first. A mismatch is the build's error, and `fluid apply` ends with exit code `1`. Nothing new was started. A running job of another config, which `fluid apply` cancels before it starts the new one, is already cancelled by then, so the stream stays stopped until the next `fluid apply`: + + ```text + Spark ingest-orders (emr-ec2): failed. Spark build ingest-orders: s3://commerce-lake/_fluid/spark/commerce.orders-stream/ingest-orders/job.json or its runner is not the one this apply compiled (another apply has changed it since); nothing was started. Run fluid apply again: it uploads them and starts the run on them + ``` + + An object that cannot be read, or is larger than 4 MiB, refuses the start in the same way. forge reads the config first and then the runner, and the message names the object it could not read: `s3://commerce-lake/_fluid/spark/commerce.orders-stream/ingest-orders/job.json cannot be read to check it before the run starts (AccessDenied); nothing was started`, or `... is larger than any artifact forge writes; nothing was started`. The word in parentheses is the code that AWS answered with, or the name of the error when AWS did not answer. A scheduler's start, `fluid apply --mode amend-and-build`, goes through the same check and fails in the same way. + +2. **The run carries the digest, and the runner refuses any other config before it reads anything.** Every runtime, local included, starts the runner with `--fluid-job-sha256 ` after the arguments that identify the run. These are the runner's arguments in the EMR on EC2 step of the example above: + + ```text + --config s3://commerce-lake/_fluid/spark/commerce.orders-stream/ingest-orders/job.json --fluid-config-hash 4db36743c76cdbdd980138a0689119de161e565181a9546b0c3ca01a76ad0f49 --fluid-source 8de3ab14e359cfad --fluid-checkpoint 1a04314328989295 --fluid-job-sha256 5316fc870295d2ff984b6d407947d73ec4d316fdab2f0f48b3f34ba483d1d9dd + ``` + + The runner computes the digest of the config it reads, and when that is not the run's, it fails with `JobConfigError` before it reads a record. A runtime that starts the job again after a failure reads the config again, and the runner checks it again: + + ```text + JobConfigError: the job config at s3://commerce-lake/_fluid/spark/commerce.orders-stream/ingest-orders/job.json is not the one fluid apply started this run on (its digest is 5316fc870295d2ff984b6d407947d73ec4d316fdab2f0f48b3f34ba483d1d9dd; the run's is 0000000000000000000000000000000000000000000000000000000000000000); nothing was read. Run fluid apply again: it starts a run on the config it uploads + ``` + + A run that someone starts by hand, without `--fluid-job-sha256`, is not checked. + +#### The digest + +The digest is the sha256 of the config's JSON in one canonical form: keys sorted, no spaces, strings in Unicode NFC (OpenTofu writes them so). Two things are not part of it as they are written. For a source that names an `mskClusterArn`, the broker list is left out, because OpenTofu fills it in at apply and MSK can change it, and a change would otherwise stop a stream's retries. The account number that OpenTofu fills into a config is the one that `fluid apply` fills into the compiled config before it compares. Every other field counts, a different `groupIdPrefix` included. + +#### What the check does not cover + +The check guards against another apply changing the config or the runner. Write access to the artifact bucket stays the boundary: someone who can write there can replace the runner after the read. The identity that runs `fluid apply` needs `s3:GetObject` on the two objects, and `kms:Decrypt` where the bucket encrypts them with a KMS key. The scheduler's policy `fluid--spark-caller` already holds `s3:GetObject` on both, but not `kms:Decrypt`: add that for the scheduler's role where the bucket encrypts them with a KMS key. + +The runner changed, so each running Spark stream restarts once on its next apply, on its checkpoint. + +The outputs are from forge-cli's reconcile step, the code `fluid apply` runs after `tofu apply`, with AWS answered by botocore's test stubs (the reads of S3 included), and from forge's runner, run as a script with a stand-in for Spark. No cluster, bucket or Spark job was involved. + ### The caller's permissions The role that runs `fluid apply`, or the scheduler's execution role (the MWAA role, say) that runs `fluid apply --mode amend-and-build` for a scheduled build, needs these actions. A scheduler's worker runs the whole apply, `tofu` included, so it also needs the apply's own permissions and the state backend, as for any scheduled build: the caller policy below covers the run only. For a build that runs to completion, the managed policy `fluid--spark-caller` holds them, scoped to the product's own application, job, cluster or virtual cluster: @@ -1016,6 +1061,8 @@ The role that runs `fluid apply`, or the scheduler's execution role (the MWAA ro Scope `iam:PassRole` to the forge roles, so that a contract cannot pass a stronger one. +*On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* A start on AWS also reads the build's job config and runner from S3, so the role needs `s3:GetObject` on both objects: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). The generated `fluid--spark-caller` policy holds it in an `Artifacts` statement, and holds no `kms:Decrypt`. + ### When a build leaves the contract *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* When a plan deletes the resources of a Spark build on AWS, because the build left the contract or the product went, `fluid apply` stops the build's runs before `tofu apply` destroys the role, the artifacts and the log group that they run with. Before this change the apply had no step for this. forge-cli's changelog says that it destroyed those and left the stream going, a writer that nothing reconciles any more, and that a destroy of a whole product failed part way, because EMR Serverless deletes only an application that is stopped or created and IAM deletes no policy that is still attached. diff --git a/docs/upgrading.md b/docs/upgrading.md index 1f17e75..ec01a2d 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1463,6 +1463,10 @@ These changes are on forge-cli `main`. The pin on this site stays at 0.32.0 unti - **Removing a Spark build on AWS needs no `--allow-data-loss`, unless the build has a dead-letter table.** The data-loss gate counted what a removed build ran with (its runner and job config objects, role and policies, log group and the key its logs are encrypted with, Glue job and connection, and the product's own EMR Serverless application) as data, so a plan that drops a build was refused unless the apply was allowed to destroy data, and that allowance also let a real data loss in the same plan through. They are now listed as removed and not gated. An output table, a bucket and a bucket's own key still count, and so does a stream's dead-letter table (`sink: table`), which goes with the build, so a plan that removes one of them needs the flag as before. A pipeline that passes it only to remove a build that has no dead-letter table no longer needs to. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract). +*On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* + +- **A Spark run starts only on the job config that `fluid apply` compiled.** Every runtime, local included, passes the config's digest at submit (`--fluid-job-sha256`), and forge's runner refuses any other config before it reads anything. On AWS, `fluid apply` first reads the job config and the runner back from S3, and starts nothing unless they are the ones this apply compiled: the build fails with `or its runner is not the one this apply compiled`, and `fluid apply` ends with exit code `1`. A running job of another config is already cancelled by then, so run `fluid apply` again: it uploads both and starts the run. The identity that runs `fluid apply` needs `s3:GetObject` on both objects (the scheduler's `fluid--spark-caller` policy already holds `s3:GetObject`, but not `kms:Decrypt`). Write access to the artifact bucket stays the boundary. The runner changed, so each running Spark stream restarts once on its next apply, on its checkpoint. See [A run starts only on the config this apply compiled](./advanced/spark-iceberg-aws.md#a-run-starts-only-on-the-config-this-apply-compiled). + ## Older releases Upgrading from before `0.15.0`? Start with the