From ad3fb9f787711d55174f735c03e064d42d6befe7 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 09:27:11 +0100 Subject: [PATCH] docs: every forge-cli change in 0.34.0 reads since 0.34.0, and the releases map is refreshed 51 markers of the form on forge-cli main, not yet in a release become since 0.34.0; the section Not in a release yet is Changes in 0.34.0; three sentences that said on main for one of these changes say 0.34.0. --- docs/advanced/airflow.md | 4 +- docs/advanced/environment-variables.md | 8 ++-- docs/advanced/spark-iceberg-aws.md | 56 +++++++++++++------------- docs/cli/apply.md | 12 +++--- docs/cli/generate.md | 4 +- docs/providers/aws.md | 2 +- docs/upgrading.md | 22 +++++----- scripts/forge-cli-releases.json | 13 +++++- 8 files changed, 65 insertions(+), 56 deletions(-) diff --git a/docs/advanced/airflow.md b/docs/advanced/airflow.md index 36aed1b..5d00ed5 100644 --- a/docs/advanced/airflow.md +++ b/docs/advanced/airflow.md @@ -78,7 +78,7 @@ In practice: - Set `FLUID_PROJECT_DIR` on the worker. Without it the task fails with `set FLUID_PROJECT_DIR on the Airflow worker to the directory that holds the product checkout`. The contract path is the path you gave the generator, relative to the working directory, or `--contract-path` when the worker's layout differs. - Put the credentials the apply needs in the worker environment. The DAG starts `fluid` with an empty environment plus the names above, so a variable that is neither on the list nor named by a `{{ env.NAME }}` or `${NAME}` placeholder in the contract, nor an `env://NAME` `secretRef`, does not arrive. Add one with `FLUID_DAG_ENV_PASSTHROUGH="NAME1 NAME2"`. - A DAG uses `max_active_runs=1` and `catchup=False`, retries from the build's `execution.retries.maxAttempts` (attempts minus one), and sets a three-hour execution timeout. -- *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* The task's `fluid apply` runs a dbt or Python build once, whichever of these its `trigger.type` is: `schedule`, `dataset`, `schedule_and_dataset`, `event`, `dependency` or `timetable`. The apply runs the build once. What decides when is the DAG for `schedule`, `dataset` and `schedule_and_dataset`, the DAG of the builds it depends on for `dependency`, and another scheduler or an event for `event` and `timetable`, which get no DAG. In 0.33.0 only `schedule` ran: a dbt or Python build of any other of these types failed with `Unknown trigger type` and exit code `1`, so the task of a `dataset` or `schedule_and_dataset` build, which this page generates a DAG for, failed. A `manual` build keeps its `iterations`, `continuous` is refused, and a `trigger.type` that is none of the names the schema knows fails with `Unknown trigger type` and the list `manual, dataset, dependency, event, schedule, schedule_and_dataset, timetable`. On 0.33.0 a `schedule_and_dataset` build printed this, and on `main` it runs once: +- *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* The task's `fluid apply` runs a dbt or Python build once, whichever of these its `trigger.type` is: `schedule`, `dataset`, `schedule_and_dataset`, `event`, `dependency` or `timetable`. The apply runs the build once. What decides when is the DAG for `schedule`, `dataset` and `schedule_and_dataset`, the DAG of the builds it depends on for `dependency`, and another scheduler or an event for `event` and `timetable`, which get no DAG. In 0.33.0 only `schedule` ran: a dbt or Python build of any other of these types failed with `Unknown trigger type` and exit code `1`, so the task of a `dataset` or `schedule_and_dataset` build, which this page generates a DAG for, failed. A `manual` build keeps its `iterations`, `continuous` is refused, and a `trigger.type` that is none of the names the schema knows fails with `Unknown trigger type` and the list `manual, dataset, dependency, event, schedule, schedule_and_dataset, timetable`. On 0.33.0 a `schedule_and_dataset` build printed this, and in 0.34.0 it runs once: ```text Trigger: schedule_and_dataset @@ -438,7 +438,7 @@ The build phase of that apply starts one run on EMR Serverless, Glue, EMR on EC2 - **Timeouts.** `execution.runtime.timeout`, an ISO-8601 duration such as `PT2H`, is how long the run may take; six hours when unset. The task's timeout is the three hours every apply DAG gets plus that wait. A value that is not a duration of at least one second is refused before `tofu apply` changes anything (`apply_spark_invalid`). - **Retries** come from `execution.retries.maxAttempts`, as for every scheduled build. -- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A scheduler task can keep it running between applies: it runs `fluid apply --mode reconcile --build-id --yes` every few minutes, and starts the stream again when it died, within its restart budget. See [Reconcile mode](../cli/apply.md#reconcile-mode). A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers). +- **A streaming build gets no DAG**, unless its `properties.stream.trigger` is `availableNow: true`. A stream runs continuously, and `fluid apply` keeps it running. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A scheduler task can keep it running between applies: it runs `fluid apply --mode reconcile --build-id --yes` every few minutes, and starts the stream again when it died, within its restart budget. See [Reconcile mode](../cli/apply.md#reconcile-mode). A build is a stream when `parameters.spark.mode` says so, or when it declares `properties.stream` and sets no mode. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* A build that depends on a stream does not wait for it; see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers). - **The worker runs the whole apply**, `tofu` included, so it needs the apply's permissions and the state backend, as for any scheduled build, plus `fluid` with boto3 (`data-product-forge[aws]` in MWAA's `requirements.txt`). The scheduler policy forge emits for a Spark product covers the run only. The operator settings (`FLUID_SPARK_*`) and the run's AWS actions are the ones [Spark to Iceberg on AWS](./spark-iceberg-aws.md#the-caller-s-permissions) lists. - A build that is not Spark gets the standard DAG, as before. A contract of dbt and Spark builds gets a DAG for each scheduled build. *(since 0.33.0, [forge-cli #966](https://github.com/Agenticstiger/forge-cli/pull/966))* Builds that `builds[].dependencies` link share one DAG instead (see [Build graphs and dataset triggers](#build-graphs-and-dataset-triggers)). diff --git a/docs/advanced/environment-variables.md b/docs/advanced/environment-variables.md index 81c65cc..3dd8d73 100644 --- a/docs/advanced/environment-variables.md +++ b/docs/advanced/environment-variables.md @@ -94,7 +94,7 @@ fluid apply runtime/plan.json --bundle runtime/bundle.tgz --env aws --yes | `FLUID_SPARK_EXECUTION_ROLE_ALLOWLIST` | The roles `parameters.spark.executionRoleArn` may name: a role ARN, or an ARN prefix ending in `/`. | | `FLUID_SPARK_CONF_ALLOWLIST` | `sparkConf` keys beyond the built-in tuning keys: a key, or a prefix ending in `*`. A key that loads code, sets credentials or moves the catalog or checkpoint is refused even when listed. | | `FLUID_SPARK_CONTRACT_SQL_ALLOWLIST` | The builds whose own `properties.sql` a Spark job may run: ``, `/`, or `/@sha256:`. *(since 0.33.0, [forge-cli #990](https://github.com/Agenticstiger/forge-cli/pull/990))* `fluid validate` reads it too, and warns only about a build it does not approve: see [Contract SQL and `fluid validate`](./spark-iceberg-aws.md#contract-sql-and-fluid-validate). | -| `FLUID_SPARK_WAIT_SECONDS` | How long a reconcile waits for a cancelled run to stop and a started one to leave its pending states. Default 900. *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* It is also how long `fluid apply` waits, for each run, for the runs and the application that a destroy would leave going: see [When a build leaves the contract](./spark-iceberg-aws.md#when-a-build-leaves-the-contract). A value that is not a positive number is ignored. | +| `FLUID_SPARK_WAIT_SECONDS` | How long a reconcile waits for a cancelled run to stop and a started one to leave its pending states. Default 900. *(since 0.34.0, [forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008))* It is also how long `fluid apply` waits, for each run, for the runs and the application that a destroy would leave going: see [When a build leaves the contract](./spark-iceberg-aws.md#when-a-build-leaves-the-contract). A value that is not a positive number is ignored. | | `FLUID_SPARK_MAVEN_REPOSITORY` | One https Maven mirror that replaces Confluent's repository in `spark.jars.repositories`. A URL with a user or password, a query, a comma, plain http or a link-local host is refused. | | `FLUID_SPARK_GLUE_JARS_URL` | `s3:////`: where the platform keeps the pinned jars of a Glue Confluent Avro build. | | `FLUID_SPARK_SUBMIT_EXECUTABLE` | The `spark-submit` a build of `parameters.spark.runtime: local` starts: an absolute path to an executable outside the project, or a name on `PATH`. Default: `spark-submit` on `PATH`. *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* [Running a build on this machine](./spark-iceberg-aws.md#operator-settings-for-a-local-run). `fluid doctor --env` lists it. | @@ -184,9 +184,9 @@ See [Airflow](./airflow.md) for which variables reach `fluid apply` on the worke | `FLUID_DBT_FORWARD_ENV` | Extra environment variable names or prefixes the dbt runner forwards. | | `DBT_EXECUTABLE` | The dbt command the runner and the `--dbt-validate` and `--dbt-tests-key auto` probes resolve first: an absolute path, a bare name on `PATH`, or a multi-token wrapper. Falls back to `dbt` on `PATH`, then the active venv. | | `FLUID_DBT_TESTS_KEY` | `auto` (default), `tests` or `data_tests`: which YAML key generated dbt tests attach under. The `--dbt-tests-key` flag wins. | -| `DBT_DOCKER_IMAGE` | A prebuilt image for the dbt container fallback. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* It must be pinned by digest (`name[:tag]@sha256:`), and a tag is refused. See [Which dbt `fluid apply` runs](../cli/generate.md#which-dbt-fluid-apply-runs). | -| `DBT_BOOTSTRAP_IMAGE` | Base image the dbt container fallback installs dbt into. Default `python:3.12-slim`. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* The default is `python:3.12-slim` pinned by digest, and an override must be pinned by digest too. | -| `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* Without it the install is the adapter's pinned package (`dbt-athena-community==1.11.1` for `athena`) plus `dbt-core==1.11.17`. It takes one or more exact requirements (`name==version`) separated by spaces or commas, and forge adds `dbt-core==1.11.17` unless the list names `dbt-core`: name it with the version you want to run another dbt-core. A range is refused. | +| `DBT_DOCKER_IMAGE` | A prebuilt image for the dbt container fallback. *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* It must be pinned by digest (`name[:tag]@sha256:`), and a tag is refused. See [Which dbt `fluid apply` runs](../cli/generate.md#which-dbt-fluid-apply-runs). | +| `DBT_BOOTSTRAP_IMAGE` | Base image the dbt container fallback installs dbt into. Default `python:3.12-slim`. *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* The default is `python:3.12-slim` pinned by digest, and an override must be pinned by digest too. | +| `DBT_ADAPTER_PACKAGE` | Overrides the pip package that fallback installs. Without it the install is `dbt-<2` plus `dbt-core<2`; setting it is how you opt in to dbt 2. *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* Without it the install is the adapter's pinned package (`dbt-athena-community==1.11.1` for `athena`) plus `dbt-core==1.11.17`. It takes one or more exact requirements (`name==version`) separated by spaces or commas, and forge adds `dbt-core==1.11.17` unless the list names `dbt-core`: name it with the version you want to run another dbt-core. A range is refused. | | `FLUID_SOURCE_DATABASE`, `FLUID_SOURCE_SCHEMA` | Read through dbt's `env_var()` in generated `sources.yml`: when set, they supply a source's database and schema before the dbt target's own. | | `FLUID_ROLLBACK_KEEP_LAST_N` | Per-product retention of `.fluid/rollback-state.json`. Default `20`. | | `FLUID_OPENLINEAGE_URL` (or the standard `OPENLINEAGE_URL`) | Base URL of an OpenLineage receiver. Without one, no lineage is emitted. Emitting needs the `lineage` extra. *(since 0.28.0, [forge-cli #837](https://github.com/Agenticstiger/forge-cli/pull/837))* | diff --git a/docs/advanced/spark-iceberg-aws.md b/docs/advanced/spark-iceberg-aws.md index 3a3fdcf..4b62606 100644 --- a/docs/advanced/spark-iceberg-aws.md +++ b/docs/advanced/spark-iceberg-aws.md @@ -169,13 +169,13 @@ An `emr-eks` build with no `releaseLabel` gets a new config hash, because its de | `watermark` | `{column, delay}`. The column must be a top-level `timestamp_ltz`: Spark 3.5 does not keep a watermark on `timestamp` (an NTZ type). | | `checkpoint` | An `s3://` prefix. It defaults to `s3:///_checkpoints///`, at the bucket root and never under the table, where Iceberg's orphan-file cleanup would delete it. When forge creates the job's role, a declared checkpoint must sit under `s3:///_checkpoints//`; one outside it is refused (`spark-checkpoint-scope`). | | `startingOffsets` | `earliest` or `latest`, for a stream with no checkpoint yet. | -| `maxOffsetsPerTrigger` | *On forge-cli `main`, not yet in a release ([forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015)).* The most Kafka records one micro-batch reads: a whole number from 1 to 9223372036854775807, split across the topic's partitions by their backlog. Unset, a batch reads everything that is waiting. A Spark stream only: managed Flink refuses the key. See [Bounding a batch](#bounding-a-batch). | +| `maxOffsetsPerTrigger` | *(since 0.34.0, [forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015))* The most Kafka records one micro-batch reads: a whole number from 1 to 9223372036854775807, split across the topic's partitions by their backlog. Unset, a batch reads everything that is waiting. A Spark stream only: managed Flink refuses the key. See [Bounding a batch](#bounding-a-batch). | | `registry` | The schema registry of an `avro-confluent` stream. On a forge Spark build only `kind: confluent` is accepted: `kind: glue` is refused. Its `auth` takes `env://` or `aws-sm://` references only. See [Schema registry](./schema-registry.md). | | `deadLetter` | *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* Where a record the stream cannot land goes, instead of failing the stream: a table beside the output (`sink: table`, the default) or a Kafka topic (`sink: topic`), with `maxRecordsBeforeAbort` (default 10000) and `alertOn: [schema_violation]`. An Avro stream is refused. See [Dead letters](#dead-letters). | ### Bounding a batch -*On forge-cli `main`, not yet in a release ([forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015)).* A stream that has been stopped, or that starts from `earliest`, finds a backlog, and without a limit its first batch reads all of it. `properties.stream.maxOffsetsPerTrigger` is the most records one micro-batch reads. forge passes it to Spark's Kafka source, which splits it across the topic's partitions by their backlog, so the stream catches up in batches of bounded size and memory. It holds for a `trigger.availableNow` run as well: the run ends once the backlog is read, in as many batches as the limit needs. +*(since 0.34.0, [forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015))* A stream that has been stopped, or that starts from `earliest`, finds a backlog, and without a limit its first batch reads all of it. `properties.stream.maxOffsetsPerTrigger` is the most records one micro-batch reads. forge passes it to Spark's Kafka source, which splits it across the topic's partitions by their backlog, so the stream catches up in batches of bounded size and memory. It holds for a `trigger.availableNow` run as well: the run ends once the backlog is read, in as many batches as the limit needs. ```yaml properties: @@ -317,13 +317,13 @@ A topic sink sends each dead letter to a topic on the cluster the stream reads, | `__connect.errors.stage` | `VALUE_CONVERTER` when the reason starts with `the value`, `KEY_CONVERTER` when it starts with `the key`, and `TRANSFORMATION` otherwise. | | `__connect.errors.exception.message` | The reason. | -*On forge-cli `main`, not yet in a release ([forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015)).* The key is where the record came from, `-@`, as in `commerce.orders-2@41874`, and the record's own key moves to one more header, which holds the bytes it was read as and not text: +*(since 0.34.0, [forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015))* The key is where the record came from, `-@`, as in `commerce.orders-2@41874`, and the record's own key moves to one more header, which holds the bytes it was read as and not text: | Header | Value | |---|---| | `fluid.dead-letter.raw-key` | The record's own key. Null for a record that had no key, and for a build that masks fields. | -The record's Kafka timestamp and `dead_lettered_at` are not sent. A topic receives each batch at least once. On `main`, a batch that is sent again after a failure carries the same keys, so a consumer drops repeats by key, and on a topic with `cleanup.policy=compact` a record sent twice is kept once when Kafka compacts the log. A record with a null value is a delete marker on a compacted topic ([Kafka's log compaction](https://kafka.apache.org/documentation/#compaction)). The dead letters of a [masked build](#masked-builds) have no value, and neither has the dead letter of a source record that had none, so send those to a topic that deletes by retention. +The record's Kafka timestamp and `dead_lettered_at` are not sent. A topic receives each batch at least once. In 0.34.0, a batch that is sent again after a failure carries the same keys, so a consumer drops repeats by key, and on a topic with `cleanup.policy=compact` a record sent twice is kept once when Kafka compacts the log. A record with a null value is a delete marker on a compacted topic ([Kafka's log compaction](https://kafka.apache.org/documentation/#compaction)). The dead letters of a [masked build](#masked-builds) have no value, and neither has the dead letter of a source record that had none, so send those to a topic that deletes by retention. #### How the records land @@ -346,11 +346,11 @@ With `alertOn: [schema_violation]`, the job writes a line like this one to its s {"forge_spark_runner": "dead_letter_alert", "reason": "schema_violation", "records": 600} ``` -`fluid apply` treats the stopped job as a failed run, like any other (on EMR Serverless, after the retries that [the reconcile table](#starting-and-reconciling-runs) describes). With the same config hash it does not start the job again. It names the run and its state, and the message says: `It is not started again with the same config, which would fail the same way: fix the cause and change the build, and fluid apply starts the new config`. A change to the build is a change to the config hash, and a new `maxRecordsBeforeAbort` is one: `fluid plan` writes a new `configHash` for it in the plan's `spark.reconcile` action. The next `fluid apply` then starts the job on the same checkpoint. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* On EMR on EC2, Glue and a local build, a job that stopped at the limit is a stream that died, and `fluid apply` and `--mode reconcile` start it again on its checkpoint within the [restart budget](#the-restart-budget). The count belongs to the job, so it starts at 0 again in each restart, and the budget bounds the restarts. +`fluid apply` treats the stopped job as a failed run, like any other (on EMR Serverless, after the retries that [the reconcile table](#starting-and-reconciling-runs) describes). With the same config hash it does not start the job again. It names the run and its state, and the message says: `It is not started again with the same config, which would fail the same way: fix the cause and change the build, and fluid apply starts the new config`. A change to the build is a change to the config hash, and a new `maxRecordsBeforeAbort` is one: `fluid plan` writes a new `configHash` for it in the plan's `spark.reconcile` action. The next `fluid apply` then starts the job on the same checkpoint. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* On EMR on EC2, Glue and a local build, a job that stopped at the limit is a stream that died, and `fluid apply` and `--mode reconcile` start it again on its checkpoint within the [restart budget](#the-restart-budget). The count belongs to the job, so it starts at 0 again in each restart, and the budget bounds the restarts. #### Masked builds -A build that masks a column at ingest (`maskAtIngest`, see [`maskAtIngest: hash`](#maskatingest-hash)) keeps no raw bytes, because they would hold the masked fields in clear. Its dead letters store the reason, the record's topic, partition, offset and timestamp, and `dead_lettered_at`, and `raw_value` and `raw_key` are null. A topic sink reads the same two columns, so the value it sends is null and, in 0.33.0, its key is null too, and the headers carry the rest. *On forge-cli `main`, not yet in a release ([forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015)).* The key of every dead letter is `-@`, a masked build's included, and the header `fluid.dead-letter.raw-key` is null. The topic, partition and offset find the record in Kafka while the topic still retains it. +A build that masks a column at ingest (`maskAtIngest`, see [`maskAtIngest: hash`](#maskatingest-hash)) keeps no raw bytes, because they would hold the masked fields in clear. Its dead letters store the reason, the record's topic, partition, offset and timestamp, and `dead_lettered_at`, and `raw_value` and `raw_key` are null. A topic sink reads the same two columns, so the value it sends is null and, in 0.33.0, its key is null too, and the headers carry the rest. *(since 0.34.0, [forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015))* The key of every dead letter is `-@`, a masked build's included, and the header `fluid.dead-letter.raw-key` is null. The topic, partition and offset find the record in Kafka while the topic still retains it. #### What is refused @@ -466,7 +466,7 @@ jq '.data.aws_iam_policy_document | to_entries[] | select(.key | endswith("_acce } ``` -When `binding.governance.lakeFormation` governs the output, the role also gets Lake Formation permissions on the dead-letter table: `ALTER`, `DESCRIBE`, `INSERT` and `SELECT`, and not `DELETE`, which [Lake Formation writes](#lake-formation-writes) grants for a batch overwrite. *On forge-cli `main`, not yet in a release ([forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010)).* An output with no `governance.lakeFormation` block gets the same grant where the account's Lake Formation settings govern new tables: see [A job's grant on its own table](#a-job-s-grant-on-its-own-table). On `emr-ec2` forge emits the policy for the cluster's owner to attach and no role, so it emits no Lake Formation permission there. From `fluid generate iac` on the contract with `governance: {lakeFormation: {grants: []}}`: +When `binding.governance.lakeFormation` governs the output, the role also gets Lake Formation permissions on the dead-letter table: `ALTER`, `DESCRIBE`, `INSERT` and `SELECT`, and not `DELETE`, which [Lake Formation writes](#lake-formation-writes) grants for a batch overwrite. *(since 0.34.0, [forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010))* An output with no `governance.lakeFormation` block gets the same grant where the account's Lake Formation settings govern new tables: see [A job's grant on its own table](#a-job-s-grant-on-its-own-table). On `emr-ec2` forge emits the policy for the cluster's owner to attach and no role, so it emits no Lake Formation permission there. From `fluid generate iac` on the contract with `governance: {lakeFormation: {grants: []}}`: ```bash jq '.resource.aws_lakeformation_permissions | with_entries(select(.key | contains("dead_letters")))' iac-governed/main.tf.json @@ -600,8 +600,8 @@ A mapped column can set `maskAtIngest: hash`. The column is the HMAC-SHA256 of i | Artifacts | `aws_s3_object` in the output's bucket. `job.json` at `_fluid/spark///job.json`, and the runner at `_fluid/spark///runner/<16 hex digits of its sha256>/forge_spark_runner.py`. A new runner is a new key. | | Logs | A CloudWatch log group `/aws/fluid/spark//`, encrypted with a per-product KMS key that only CloudWatch Logs may use, for this product's groups. | | The job's role | Unless the contract names one. Least privilege: S3 read and write on the data and checkpoint prefixes, read on the artifacts, the Glue calls on the output, KMS through S3 and Secrets Manager only, MSK IAM on the cluster, topic and consumer group, `lakeformation:GetDataAccess`, the mask key and a registry secret, and its log group. | -| A dead-letter table | *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* For a stream with `deadLetter` and `sink: table`: an Iceberg v2 `aws_glue_catalog_table` beside the output's table, with its S3 prefix, Glue table and, when Lake Formation governs the output, Lake Formation permissions added to the job's role. *On forge-cli `main`, not yet in a release ([forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010)).* Without a `governance.lakeFormation` block, those permissions are added where the account's Lake Formation settings govern new tables: see [A job's grant on its own table](#a-job-s-grant-on-its-own-table). A `sink: topic` emits no table: on MSK with IAM the job's role gets the permissions to write the topic. See [What OpenTofu emits for dead letters](#what-opentofu-emits-for-dead-letters). | -| A caller policy | For a build that runs to completion (batch, or a stream with `availableNow`): the managed policy `fluid--spark-caller`, for the platform to attach to the scheduler's role. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A stream that runs continuously gets it too, so that a keepalive can run `fluid apply --mode reconcile` from the scheduler's role. | +| A dead-letter table | *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* For a stream with `deadLetter` and `sink: table`: an Iceberg v2 `aws_glue_catalog_table` beside the output's table, with its S3 prefix, Glue table and, when Lake Formation governs the output, Lake Formation permissions added to the job's role. *(since 0.34.0, [forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010))* Without a `governance.lakeFormation` block, those permissions are added where the account's Lake Formation settings govern new tables: see [A job's grant on its own table](#a-job-s-grant-on-its-own-table). A `sink: topic` emits no table: on MSK with IAM the job's role gets the permissions to write the topic. See [What OpenTofu emits for dead letters](#what-opentofu-emits-for-dead-letters). | +| A caller policy | For a build that runs to completion (batch, or a stream with `availableNow`): the managed policy `fluid--spark-caller`, for the platform to attach to the scheduler's role. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A stream that runs continuously gets it too, so that a keepalive can run `fluid apply --mode reconcile` from the scheduler's role. | Per runtime: @@ -780,7 +780,7 @@ spark.sql.shuffle.partitions=16 1a04314328989295 ``` -*On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* The argv of a run on `main` ends with two more lines, `--fluid-job-sha256` and the digest of the job config that forge wrote under the root, and the runner refuses any other config before it reads anything: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). These are the two lines in a run of the same stream on `main`: +*(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014))* The argv of a run ends with two more lines, `--fluid-job-sha256` and the digest of the job config that forge wrote under the root, and the runner refuses any other config before it reads anything: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). These are the two lines in a run of the same stream on `main`: ```text --fluid-job-sha256 @@ -882,7 +882,7 @@ A local stream follows the `spark.reconcile` action of [Starting and reconciling | A stream that failed with this config | An error that names the run and its log, and no new start. Change the build, and apply starts the new config. | | A stream of a build that the contract no longer declares | Stops it: `stopped run CANCELLED`. A build moved to an AWS runtime keeps its local runs, because an apply of AWS builds never stops them. | -*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* The rows about a stream stopped from outside forge and a stream that failed change. A stream that failed, one whose supervisor got a signal from outside forge (a shutdown of the machine, say), and one whose processes ended without recording how (`ended LOST`) are a death: `fluid apply` and `--mode reconcile` start it again on its checkpoint within the [restart budget](#the-restart-budget), and count each restart. A spent budget is the error. Only a run that was cancelled through forge is left stopped, and only by `--mode reconcile`. +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* The rows about a stream stopped from outside forge and a stream that failed change. A stream that failed, one whose supervisor got a signal from outside forge (a shutdown of the machine, say), and one whose processes ended without recording how (`ended LOST`) are a death: `fluid apply` and `--mode reconcile` start it again on its checkpoint within the [restart budget](#the-restart-budget), and count each restart. A spent budget is the error. Only a run that was cancelled through forge is left stopped, and only by `--mode reconcile`. A stop sends SIGTERM to the run's process group, and SIGKILL after a grace period of 60 seconds. forge waits for the run to end for as long as `FLUID_SPARK_WAIT_SECONDS` says. A new topic or cluster on a checkpoint that an earlier run read is refused before anything is stopped, as on AWS. @@ -904,7 +904,7 @@ Spark ingest-orders (local): failed run 20261119T060909Z-89c3a970 FAILED on this The two lines of that message, `the stream failed` and the exit code, come from a stand-in `spark-submit` that exits 1; the path is shortened to ``. -*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A failed stream is started again on its checkpoint, within the [restart budget](#the-restart-budget), and a message like the one above is the error for a spent budget. It ends with the same second way: `when the cause was this machine's (spark-submit, its properties file), remove /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs (the build's run history here) and fluid apply starts it again`. The run directory it names is the whole history, which is what the budget is counted from. +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A failed stream is started again on its checkpoint, within the [restart budget](#the-restart-budget), and a message like the one above is the error for a spent budget. It ends with the same second way: `when the cause was this machine's (spark-submit, its properties file), remove /.fluid/commerce.orders-stream/runtime/spark/ingest-orders/runs (the build's run history here) and fluid apply starts it again`. The run directory it names is the whole history, which is what the budget is counted from. **A dry run.** `fluid apply --dry-run` starts and stops nothing and creates no run directory. It prints `would be reconciled to config ` for a stream, and `would stop ` for a run the contract no longer runs here: @@ -1019,8 +1019,8 @@ After `tofu apply`, `fluid apply` reconciles each streaming build: | A running job with this hash | Nothing. | | A running job with another hash | Cancels it, waits for it to stop, and starts the new config on the **same checkpoint**, so it resumes where the old one stopped. | | A new topic or cluster, with a checkpoint an earlier run read | Refuses, before cancelling anything. A checkpoint holds the offsets of one topic on one cluster. To read the new stream, move `properties.stream.checkpoint` to a new prefix: the job then starts from `startingOffsets`. | -| No running job | Starts one, unless the last run with this hash failed. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* Unless it failed where the service's own retries are spent (EMR Serverless and EMR on EKS) or the [restart budget](#the-restart-budget) is spent: see the next row. | -| A last run that failed | An error that names the run and its reason. On EMR Serverless a streaming job retries until it fails more than `maxFailedAttemptsPerHour` times in an hour, and then stops, so starting it again would fail the same way. Fix the cause, then change the build: `fluid apply` starts it with the new config. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* On EMR on EC2, Glue and a local build the stream is started again on its checkpoint within the [restart budget](#the-restart-budget), and the error is for a spent budget. EMR Serverless and EMR on EKS keep the error, as their own retries come first. | +| No running job | Starts one, unless the last run with this hash failed. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* Unless it failed where the service's own retries are spent (EMR Serverless and EMR on EKS) or the [restart budget](#the-restart-budget) is spent: see the next row. | +| A last run that failed | An error that names the run and its reason. On EMR Serverless a streaming job retries until it fails more than `maxFailedAttemptsPerHour` times in an hour, and then stops, so starting it again would fail the same way. Fix the cause, then change the build: `fluid apply` starts it with the new config. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* On EMR on EC2, Glue and a local build the stream is started again on its checkpoint within the [restart budget](#the-restart-budget), and the error is for a spent budget. EMR Serverless and EMR on EKS keep the error, as their own retries come first. | | A second active copy of the current config | Cancels it. | | A stream moved to `trigger.availableNow` | Cancels its old continuous job. Its scheduler starts it from now on. | @@ -1036,7 +1036,7 @@ Per runtime, the start is: ### The restart budget -*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* A stream that died is started again on its checkpoint, within a budget of restarts per rolling hour. A plain `fluid apply` follows the rule, and so does [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode), which a keepalive runs. In 0.33.0 a stream whose last run ended `FAILED` was an error until the build changed, and `--mode reconcile` did not exist. +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* A stream that died is started again on its checkpoint, within a budget of restarts per rolling hour. A plain `fluid apply` follows the rule, and so does [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode), which a keepalive runs. In 0.33.0 a stream whose last run ended `FAILED` was an error until the build changed, and `--mode reconcile` did not exist. - **The budget** is `execution.retries.maxAttempts` restarts per rolling hour: 5 when it is unset, and at most 10, the numbers that EMR Serverless applies to its own streaming retries. - **A death** is a run that ended `FAILED` or `INTERRUPTED` (an EMR step that died with its node), was lost with its machine, node or cluster, or was stopped by a signal from outside forge (a local run that was killed, or whose machine shut down), and a stream that stopped by itself. A local run whose supervisor got a signal from outside forge, a shutdown of the machine say, reads as failed now, and counts as a death: 0.33.0 read it as cancelled and started it again. A job killed on its own (`FAILED (exit -9)`) was failed in 0.33.0 too. @@ -1099,7 +1099,7 @@ The outputs are from forge-cli's reconcile step. The EMR on EC2 ones are the cod *(since 0.33.0, [forge-cli #1005](https://github.com/Agenticstiger/forge-cli/pull/1005))* -A step on an `emr-ec2` cluster runs as the role of the cluster's instance profile, which the cluster's owner holds. Forge does not change that role: it belongs to the cluster's owner. `fluid apply` creates the build's least-privilege policy (the managed `aws_iam_policy` in [What forge provisions](#what-forge-provisions)), and the owner attaches it. On 0.32.0 the first `fluid apply` started the stream before anyone could attach the policy, so, unless the role already held the job's permissions, the step failed at once. In 0.33.0 a stream that failed is not started again with the same config. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* It is started again on its checkpoint, within the [restart budget](#the-restart-budget). +A step on an `emr-ec2` cluster runs as the role of the cluster's instance profile, which the cluster's owner holds. Forge does not change that role: it belongs to the cluster's owner. `fluid apply` creates the build's least-privilege policy (the managed `aws_iam_policy` in [What forge provisions](#what-forge-provisions)), and the owner attaches it. On 0.32.0 the first `fluid apply` started the stream before anyone could attach the policy, so, unless the role already held the job's permissions, the step failed at once. In 0.33.0 a stream that failed is not started again with the same config. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* It is started again on its checkpoint, within the [restart budget](#the-restart-budget). Before it starts a step, `fluid apply` now reads the cluster's instance profile and the policies attached to the role in it, and starts nothing until the build's policy is among them. The refusal is the build's error and `fluid apply` ends with exit code `1`. The infrastructure is applied by then, so the policy exists, and the line gives the command that attaches it: @@ -1117,7 +1117,7 @@ The output above is from forge-cli's reconcile step, the code `fluid apply` runs ### A run starts only on the config this apply compiled -*On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* A run's job config is an object in S3, which the runtime reads when the run starts and again each time it starts the job again after a failure. Until this change nothing checked what the object held: a config that another `fluid apply` had put there since this one compiled its runs would run under the config hash of this apply's runs, and a later `fluid apply` would take the hash for the config. Now a run starts only on the config that `fluid apply` compiled, in two steps. +*(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014))* A run's job config is an object in S3, which the runtime reads when the run starts and again each time it starts the job again after a failure. Until this change nothing checked what the object held: a config that another `fluid apply` had put there since this one compiled its runs would run under the config hash of this apply's runs, and a later `fluid apply` would take the hash for the config. Now a run starts only on the config that `fluid apply` compiled, in two steps. 1. **`fluid apply` reads the job config and the runner back from S3 before it starts a run, and starts nothing unless both are the ones it compiled.** This is for the runs on AWS: EMR Serverless, Glue, EMR on EC2 and EMR on EKS. The job config counts when its [digest](#the-digest) is the compiled config's. The runner counts when it is, byte for byte, the runner that this forge-cli ships. On EMR on EC2 the [check that the policy is attached](#an-emr-on-ec2-step-starts-once-its-policy-is-attached) comes first. A mismatch is the build's error, and `fluid apply` ends with exit code `1`. Nothing new was started. A running job of another config, which `fluid apply` cancels before it starts the new one, is already cancelled by then, so the stream stays stopped until the next `fluid apply`: @@ -1166,15 +1166,15 @@ The role that runs `fluid apply`, or the scheduler's execution role (the MWAA ro Scope `iam:PassRole` to the forge roles, so that a contract cannot pass a stronger one. -*On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* A start on AWS also reads the build's job config and runner from S3, so the role needs `s3:GetObject` on both objects: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). The generated `fluid--spark-caller` policy holds it in an `Artifacts` statement, and holds no `kms:Decrypt`. +*(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014))* A start on AWS also reads the build's job config and runner from S3, so the role needs `s3:GetObject` on both objects: see [A run starts only on the config this apply compiled](#a-run-starts-only-on-the-config-this-apply-compiled). The generated `fluid--spark-caller` policy holds it in an `Artifacts` statement, and holds no `kms:Decrypt`. -*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* The policy also covers the product's always-on streams: it lets a role start, watch and stop their runs and read their job config and runner, so that a keepalive can run [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode) from the scheduler's role (the MWAA role, say). It has one statement for each runtime, named `Run`, that lists the runtime's actions once for every target of the product's builds on it, one `PassRole` statement for each runtime that passes a role (EMR Serverless and EMR on EKS), and one `Artifacts` statement. Two more appear where they apply: `StepRoleEmrEc2`, the two IAM reads of an EMR on EC2 start, and `FindApplication`, `emr-serverless:ListApplications` on `*`, for an EMR Serverless build that names no `applicationId`. A managed policy holds at most 6,144 characters, and a statement for each build reached that limit at about six builds. Each build's target, role and artifacts are still named in them. A bucket that encrypts the artifacts with a KMS key also needs `kms:Decrypt` on its key for the scheduler's role, which forge does not grant. `--mode reconcile` reads the uploaded job config on every run, so a role that lacks `s3:GetObject` on it fails while the stream is healthy. +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* The policy also covers the product's always-on streams: it lets a role start, watch and stop their runs and read their job config and runner, so that a keepalive can run [`fluid apply --mode reconcile`](../cli/apply.md#reconcile-mode) from the scheduler's role (the MWAA role, say). It has one statement for each runtime, named `Run`, that lists the runtime's actions once for every target of the product's builds on it, one `PassRole` statement for each runtime that passes a role (EMR Serverless and EMR on EKS), and one `Artifacts` statement. Two more appear where they apply: `StepRoleEmrEc2`, the two IAM reads of an EMR on EC2 start, and `FindApplication`, `emr-serverless:ListApplications` on `*`, for an EMR Serverless build that names no `applicationId`. A managed policy holds at most 6,144 characters, and a statement for each build reached that limit at about six builds. Each build's target, role and artifacts are still named in them. A bucket that encrypts the artifacts with a KMS key also needs `kms:Decrypt` on its key for the scheduler's role, which forge does not grant. `--mode reconcile` reads the uploaded job config on every run, so a role that lacks `s3:GetObject` on it fails while the stream is healthy. ### When a build leaves the contract -*On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* When a plan deletes the resources of a Spark build on AWS, because the build left the contract or the product went, `fluid apply` stops the build's runs before `tofu apply` destroys the role, the artifacts and the log group that they run with. Before this change the apply had no step for this. forge-cli's changelog says that it destroyed those and left the stream going, a writer that nothing reconciles any more, and that a destroy of a whole product failed part way, because EMR Serverless deletes only an application that is stopped or created and IAM deletes no policy that is still attached. +*(since 0.34.0, [forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008))* When a plan deletes the resources of a Spark build on AWS, because the build left the contract or the product went, `fluid apply` stops the build's runs before `tofu apply` destroys the role, the artifacts and the log group that they run with. Before this change the apply had no step for this. forge-cli's changelog says that it destroyed those and left the stream going, a writer that nothing reconciles any more, and that a destroy of a whole product failed part way, because EMR Serverless deletes only an application that is stopped or created and IAM deletes no policy that is still attached. -*On forge-cli `main`, not yet in a release ([forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007)).* Removing a build needs no `--allow-data-loss`, unless the build has a dead-letter table. What a Spark build runs with holds no data, so the [data-loss gate](../cli/apply.md#opentofu-data-loss-gate) does not count it: the runner and the job config in the artifact bucket, the copies of the jars that a Glue build loads, the build's log group, its role and the role's policies (for an EMR on EC2 build, its job policy), the Glue job and its connection, the product's own EMR Serverless application, the scheduler policy `fluid--spark-caller`, the key that encrypts the product's Spark log groups, and the role that a registered Lake Formation location is written through. [What forge provisions](#what-forge-provisions), [Avro on Glue](#avro-on-glue) and [Lake Formation writes](#lake-formation-writes) say which of them a build gets. forge-cli's changelog says that the gate counted all of them as data, so dropping a build from a contract was refused unless the apply was allowed to destroy data, and that allowance let a real data loss in the same plan through. +*(since 0.34.0, [forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007))* Removing a build needs no `--allow-data-loss`, unless the build has a dead-letter table. What a Spark build runs with holds no data, so the [data-loss gate](../cli/apply.md#opentofu-data-loss-gate) does not count it: the runner and the job config in the artifact bucket, the copies of the jars that a Glue build loads, the build's log group, its role and the role's policies (for an EMR on EC2 build, its job policy), the Glue job and its connection, the product's own EMR Serverless application, the scheduler policy `fluid--spark-caller`, the key that encrypts the product's Spark log groups, and the role that a registered Lake Formation location is written through. [What forge provisions](#what-forge-provisions), [Avro on Glue](#avro-on-glue) and [Lake Formation writes](#lake-formation-writes) say which of them a build gets. forge-cli's changelog says that the gate counted all of them as data, so dropping a build from a contract was refused unless the apply was allowed to destroy data, and that allowance let a real data loss in the same plan through. `fluid plan --tofu`, `fluid apply --dry-run` and `fluid apply` list them as removed and not gated. For the example contract's build on EMR on EC2, the line reads: @@ -1206,7 +1206,7 @@ After the gate, `fluid apply` reads the plan's deletes, and before `tofu apply` Spark ingest-orders (emr-ec2): stopped on EMR cluster j-1ABCDEF. cancelled s-1 before the apply destroys what it runs with ``` -3. **It empties and stops the product's own EMR Serverless application, when that goes.** Every active run in it is cancelled, whichever build started it, and then the application is stopped and waited for, unless it is already stopped or created. *On forge-cli `main`, not yet in a release ([forge-cli #1017](https://github.com/Agenticstiger/forge-cli/pull/1017)).* An application that is still `CREATING` or `STARTING` is waited for until it settles first: one that has been created is deleted as it is, and one that has started is stopped. One still moving after the wait refuses the apply. An application that the contract only names is not one the plan deletes, so it is left as it is. +3. **It empties and stops the product's own EMR Serverless application, when that goes.** Every active run in it is cancelled, whichever build started it, and then the application is stopped and waited for, unless it is already stopped or created. *(since 0.34.0, [forge-cli #1017](https://github.com/Agenticstiger/forge-cli/pull/1017))* An application that is still `CREATING` or `STARTING` is waited for until it settles first: one that has been created is deleted as it is, and one that has started is stopped. One still moving after the wait refuses the apply. An application that the contract only names is not one the plan deletes, so it is left as it is. ```text EMR Serverless application fluid-commerce.orders-stream-spark (00fabc123def): cancelled jr1 before the apply destroys what it runs with; stopped the application. @@ -1250,7 +1250,7 @@ What forge does with the cluster decides what the layout must protect: - **The driver is the step's own process.** forge starts the stream with `spark-submit --deploy-mode client`, so that cancelling the step stops the job. AWS's Spark step guide says client mode launches the driver on the cluster's primary instance. - **The checkpoint is in S3, not on the cluster.** It defaults to `s3:///_checkpoints///`, so losing nodes loses no offsets. -- **A step that ends `FAILED` or `INTERRUPTED` counts as failed, and `fluid apply` does not start it again with the same config.** It names the run and its state and starts nothing, because the same config would fail the same way (see [Starting and reconciling runs](#starting-and-reconciling-runs)). To bring the stream back, change the build. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* `fluid apply` and `--mode reconcile` start such a step again on the stream's checkpoint, within the [restart budget](#the-restart-budget): an `INTERRUPTED` step, which died with its node, is a restart like any other, and the error is for a spent budget. After a lost cluster, launch a replacement with forge's policy attached to its instance-profile role, and set `clusterId` to it, with its `emr-ec2:` entry in `FLUID_SPARK_TARGET_ALLOWLIST`: the target is part of the config hash, so `fluid apply` starts the stream on the same checkpoint. +- **A step that ends `FAILED` or `INTERRUPTED` counts as failed, and `fluid apply` does not start it again with the same config.** It names the run and its state and starts nothing, because the same config would fail the same way (see [Starting and reconciling runs](#starting-and-reconciling-runs)). To bring the stream back, change the build. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* `fluid apply` and `--mode reconcile` start such a step again on the stream's checkpoint, within the [restart budget](#the-restart-budget): an `INTERRUPTED` step, which died with its node, is a restart like any other, and the error is for a spent budget. After a lost cluster, launch a replacement with forge's policy attached to its instance-profile role, and set `clusterId` to it, with its `emr-ec2:` entry in `FLUID_SPARK_TARGET_ALLOWLIST`: the target is part of the config hash, so `fluid apply` starts the stream on the same checkpoint. The checklist: @@ -1258,7 +1258,7 @@ The checklist: 2. **Spot task fleets, with `price-capacity-optimized` allocation and diverse instance types.** Use instance fleets, and set the task fleet's Spot `AllocationStrategy` to `price-capacity-optimized`: AWS recommends it, and it is the default from EMR 6.10.0. List several instance types: when EC2 reclaims a Spot instance, EMR tries to replace it with any type on the list. Offer subnets in several Availability Zones too. EMR picks the zone it judges best for the fleet when it creates the cluster, and the cluster's instances then run in that one zone. 3. **YARN application masters pinned to On-Demand (EMR 7.2.0 and later).** In the cluster's `yarn-site` classification, set `yarn.node-labels.enabled` to `true` and `yarn.node-labels.am.default-node-label-expression` to `ON_DEMAND`. AWS warns that an application master on a Spot node can mean "immediate shutdown of the entire YARN application". Set it when you create the cluster: EMR does not add node labels when you reconfigure a running one. AWS describes this setting for clusters that use managed scaling, and its blog post shows instance groups. The example below uses instance fleets and no managed scaling, so check on the running cluster that YARN lists an `ON_DEMAND` node label. Settings that start with `spark.yarn.` are refused in `sparkConf` even when the operator lists them, so this belongs to the cluster and not to the contract. 4. **A minimum On-Demand executor baseline for streams.** Size the On-Demand capacity (the core nodes, and any On-Demand part of a task fleet) for the executors the stream needs to keep committing batches when every Spot node is gone, and tell Spark that number in `sparkConf`: `spark.dynamicAllocation.minExecutors` with dynamic allocation, or `spark.executor.instances` without it. Both are allowed without an operator entry, and both fix a count, not where the executors run. When Spot nodes are reclaimed, YARN can place a replacement executor only on the nodes that remain. Where YARN puts executors while Spot nodes exist depends on the cluster's node labels, so look at where they ran in the interruption test (item 6). -5. **Kafka retention at least as long as the longest downtime you tolerate.** A restarted stream resumes from the offsets in its checkpoint, so the topic must still hold them. Count the whole outage: detecting the loss, replacing the cluster, starting the step, and catching up. Set the topic's `retention.ms` (Kafka's default is 7 days) to at least that, and leave `retention.bytes` unset (`-1`) or size it for the same window, because a topic discards a segment at whichever limit it reaches first. forge's runner leaves Spark's Kafka option `failOnDataLoss` at its default, `true`, so a stream whose checkpointed offsets have aged out fails and does not skip records. *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* The same holds for a batch that a restart replays into a [dead-letter table](#how-the-records-land) or into the output through Iceberg's streaming sink: each is appended only once while the snapshot that records the batch is still in the table, so run `expire_snapshots` with a retention longer than the longest downtime you tolerate. *On forge-cli `main`, not yet in a release ([forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015)).* The runner sets `failOnDataLoss` to `true` on every Kafka source, so it no longer depends on Spark's default. Offsets that the topic no longer holds, because retention passed them while the stream was stopped or because the topic was recreated, fail the stream with a Spark error whose message contains `Some data may have been lost because they are not available in Kafka any more`. +5. **Kafka retention at least as long as the longest downtime you tolerate.** A restarted stream resumes from the offsets in its checkpoint, so the topic must still hold them. Count the whole outage: detecting the loss, replacing the cluster, starting the step, and catching up. Set the topic's `retention.ms` (Kafka's default is 7 days) to at least that, and leave `retention.bytes` unset (`-1`) or size it for the same window, because a topic discards a segment at whichever limit it reaches first. forge's runner leaves Spark's Kafka option `failOnDataLoss` at its default, `true`, so a stream whose checkpointed offsets have aged out fails and does not skip records. *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* The same holds for a batch that a restart replays into a [dead-letter table](#how-the-records-land) or into the output through Iceberg's streaming sink: each is appended only once while the snapshot that records the batch is still in the table, so run `expire_snapshots` with a retention longer than the longest downtime you tolerate. *(since 0.34.0, [forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015))* The runner sets `failOnDataLoss` to `true` on every Kafka source, so it no longer depends on Spark's default. Offsets that the topic no longer holds, because retention passed them while the stream was stopped or because the topic was recreated, fail the stream with a Spark error whose message contains `Some data may have been lost because they are not available in Kafka any more`. 6. **Spark decommissioning only after a measured interruption test.** Leave Spark's own decommissioning off until a test shows that it helps. - **What is already there.** EC2 sends a Spot interruption notice two minutes before it reclaims an instance, on a best-effort basis. EMR on EC2 has its own decommissioning for Spark, built on YARN's. Spark schedules no new tasks on a node that YARN is decommissioning, and lets the running tasks finish (`spark.blacklist.decommissioning.enabled`, `true` by default). `spark.decommissioning.timeout.threshold` (20 seconds by default, EMR 5.11.0 and later) is for Spot: a Spot instance decommissions within 20 seconds whatever the YARN decommissioning timeout is, and a node that will decommission within the threshold is deny-listed and cleaned up without waiting. - **What is separate.** Spark's own `spark.decommission.enabled` (`false` by default since Spark 3.1.0) and its `spark.storage.decommission.*` settings. A contract may set them only when the operator lists them in `FLUID_SPARK_CONF_ALLOWLIST`. @@ -1404,7 +1404,7 @@ In the account's `-fluid-data` fallback bucket, and in a `shared` packa ### A job's grant on its own table -*On forge-cli `main`, not yet in a release ([forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010)).* Up to 0.33.0 forge granted a job's role Lake Formation access to its table only for an output whose binding declares a non-empty `governance.lakeFormation` block. In an account that removed the `IAM_ALLOWED_PRINCIPALS` default for new tables, as Lake Formation advises, a stream with no such block could not see its own table. forge now emits the grant for such an output as well, when the output is a Glue table: `ALTER`, `DESCRIBE`, `INSERT` and `SELECT` on the output's table, with `DELETE` for a batch job, and `ALTER`, `DESCRIBE`, `INSERT` and `SELECT` on the dead-letter table of a stream beside it. The module is the same text in every account. The plan decides whether each grant exists: the module reads the account's Lake Formation settings, and the grant has a `count` of 1 when none of the settings' default permissions for new tables names `IAM_ALLOWED_PRINCIPALS`, and 0 when one does. An account that keeps that default creates nothing new. This is what `fluid generate iac` emits for the example contract on EMR Serverless: +*(since 0.34.0, [forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010))* Up to 0.33.0 forge granted a job's role Lake Formation access to its table only for an output whose binding declares a non-empty `governance.lakeFormation` block. In an account that removed the `IAM_ALLOWED_PRINCIPALS` default for new tables, as Lake Formation advises, a stream with no such block could not see its own table. forge now emits the grant for such an output as well, when the output is a Glue table: `ALTER`, `DESCRIBE`, `INSERT` and `SELECT` on the output's table, with `DELETE` for a batch job, and `ALTER`, `DESCRIBE`, `INSERT` and `SELECT` on the dead-letter table of a stream beside it. The module is the same text in every account. The plan decides whether each grant exists: the module reads the account's Lake Formation settings, and the grant has a `count` of 1 when none of the settings' default permissions for new tables names `IAM_ALLOWED_PRINCIPALS`, and 0 when one does. An account that keeps that default creates nothing new. This is what `fluid generate iac` emits for the example contract on EMR Serverless: ```json { @@ -1433,7 +1433,7 @@ In the account's `-fluid-data` fallback bucket, and in a `shared` packa An output with a non-empty `governance.lakeFormation` block keeps its grants as they are, and no settings are read for it. An EMR on EC2 job runs as the cluster's own role, which forge does not know, so it gets no grant and no settings are read. -*On forge-cli `main`, not yet in a release ([forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010)).* The identity that runs `fluid plan` and `fluid apply` needs `lakeformation:GetDataLakeSettings`, for a Spark build on any runtime but EMR on EC2 whose output is a Glue table with no `governance.lakeFormation` block. The grant goes to the job's role whether forge creates it or the contract names it. +*(since 0.34.0, [forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010))* The identity that runs `fluid plan` and `fluid apply` needs `lakeformation:GetDataLakeSettings`, for a Spark build on any runtime but EMR on EC2 whose output is a Glue table with no `governance.lakeFormation` block. The grant goes to the job's role whether forge creates it or the contract names it. ## Errors @@ -1442,8 +1442,8 @@ An output with a non-empty `governance.lakeFormation` block keeps its grants as | `spark-emr-application-shared` | An EMR Serverless build with no `applicationId` and no `emrApplication: isolated` | | `spark-checkpoint-scope` | A declared checkpoint outside `s3:///_checkpoints//`, while forge creates the role | | `spark-glue-jars-unset`, `spark-glue-jars-url` | A Glue Avro build, and `FLUID_SPARK_GLUE_JARS_URL` is unset or malformed | -| `spark-policy-attached` | *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* The plan destroys an IAM policy that is still attached: an EMR on EC2 build's job policy or the product's scheduler policy. The error is `opentofu_destroy_preparation_failed`. See [When a build leaves the contract](#when-a-build-leaves-the-contract). | -| `spark-destroy-runs-going` | *On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* A run of a build that goes, or the product's own application, is still going after `FLUID_SPARK_WAIT_SECONDS`, or AWS refused to list, cancel or stop it. The error is `opentofu_destroy_preparation_failed`. See [When a build leaves the contract](#when-a-build-leaves-the-contract). | +| `spark-policy-attached` | *(since 0.34.0, [forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008))* The plan destroys an IAM policy that is still attached: an EMR on EC2 build's job policy or the product's scheduler policy. The error is `opentofu_destroy_preparation_failed`. See [When a build leaves the contract](#when-a-build-leaves-the-contract). | +| `spark-destroy-runs-going` | *(since 0.34.0, [forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008))* A run of a build that goes, or the product's own application, is still going after `FLUID_SPARK_WAIT_SECONDS`, or AWS refused to list, cancel or stop it. The error is `opentofu_destroy_preparation_failed`. See [When a build leaves the contract](#when-a-build-leaves-the-contract). | | `spark-dead-letters-table-unsupported` | *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* A `deadLetter` table sink beside an output at its bucket's root, beside a location forge registers with Lake Formation, or beside an output whose lifecycle expires its objects. See [What is refused](#what-is-refused). | | `apply_spark_refused` | `fluid apply` refuses a Spark build before `tofu apply` changes anything when an operator's list does not allow what the build names: an MSK cluster that `FLUID_SPARK_TARGET_ALLOWLIST` does not list, a host, a secret reference, or a `sparkConf` key. *(since 0.33.0, [forge-cli #975](https://github.com/Agenticstiger/forge-cli/pull/975))* It also refuses a `deadLetter` topic that `FLUID_SPARK_TARGET_ALLOWLIST` does not list (see [Listing the topic](#listing-the-topic)), and, for a local build, a namespace that `FLUID_SPARK_LOCAL_NAMESPACE_ALLOWLIST` does not list (see [The table the job creates](#the-table-the-job-creates)). | | `apply_spark_artifact_mismatch` | The bound plan lists other Spark runs than apply compiles | diff --git a/docs/cli/apply.md b/docs/cli/apply.md index 37f8790..99279bb 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -136,7 +136,7 @@ Two things differ between the engines, and the rest of this page says which one | `amend-and-build` | Same as `amend`, then run the contract's `builds[]` (the build runner). | | `replace` | Destructive. Requires `--allow-data-loss`. | | `replace-and-build` | Same as `replace`, then the builds with a full refresh. Destructive. | -| `reconcile` | *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* Deploys nothing: keeps the contract's always-on Spark streams running. See [Reconcile mode](#reconcile-mode). | +| `reconcile` | *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* Deploys nothing: keeps the contract's always-on Spark streams running. See [Reconcile mode](#reconcile-mode). | What each engine does with the mode: @@ -153,9 +153,9 @@ The build-augmented modes (`amend-and-build`, `replace-and-build`) run the confi amend-and-build (or replace-and-build) to run that build, or drop --build-id. ``` -Before 0.16.3 the flag was accepted and dropped, so a DDL-only apply exited 0. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* `--build-id` is accepted with `--mode reconcile` too, where it names the stream to keep running, and the hint of this refusal adds `--mode reconcile to keep that stream running`. +Before 0.16.3 the flag was accepted and dropped, so a DDL-only apply exited 0. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* `--build-id` is accepted with `--mode reconcile` too, where it names the stream to keep running, and the hint of this refusal adds `--mode reconcile to keep that stream running`. -*On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* A dbt or Python build runs once under `--mode amend-and-build` whatever its `execution.trigger.type`, `manual` and `continuous` aside: see [What the DAG does](../advanced/airflow.md#what-the-dag-does). Which dbt a dbt build runs on, and what the container fallback installs, are in [Which dbt `fluid apply` runs](./generate.md#which-dbt-fluid-apply-runs). +*(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* A dbt or Python build runs once under `--mode amend-and-build` whatever its `execution.trigger.type`, `manual` and `continuous` aside: see [What the DAG does](../advanced/airflow.md#what-the-dag-does). Which dbt a dbt build runs on, and what the container fallback installs, are in [Which dbt `fluid apply` runs](./generate.md#which-dbt-fluid-apply-runs). *(since 0.30.0, [forge-cli #861](https://github.com/Agenticstiger/forge-cli/pull/861))* On Databricks, a dbt build with `execution.runtime.platform: databricks` runs as the job the apply provisions. The build phase starts a run of the job and waits for it to end, and a failed run fails the apply. See [Running the job from `fluid apply`](../providers/databricks.md#running-the-job-from-fluid-apply). @@ -163,7 +163,7 @@ Before 0.16.3 the flag was accepted and dropped, so a DDL-only apply exited 0. * ### Reconcile mode -*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* `--mode reconcile` keeps the contract's always-on Spark streams running, and deploys nothing: it runs no `tofu`, no DDL and no build, and it takes no plan. A stream that died is started again on its checkpoint, within its [restart budget](../advanced/spark-iceberg-aws.md#the-restart-budget), when what the last `fluid apply` deployed is what the contract compiles to. A scheduler task that runs it every few minutes, a keepalive, keeps a stream going between applies: +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* `--mode reconcile` keeps the contract's always-on Spark streams running, and deploys nothing: it runs no `tofu`, no DDL and no build, and it takes no plan. A stream that died is started again on its checkpoint, within its [restart budget](../advanced/spark-iceberg-aws.md#the-restart-budget), when what the last `fluid apply` deployed is what the contract compiles to. A scheduler task that runs it every few minutes, a keepalive, keeps a stream going between applies: ```bash fluid apply contract.fluid.yaml --env prod --mode reconcile --build-id ingest-orders --yes @@ -491,7 +491,7 @@ Removing a resource that only grants access is a revocation, not a loss, and pas A Cloud KMS key's IAM grant is not exempt: without it BigQuery cannot decrypt the table. Any removal of a type not in the list counts, so removing a bucket's lifecycle or server-side-encryption configuration, or a product's KMS key, needs `--allow-data-loss` even though no object is deleted. See [AWS](../providers/aws.md). If the plan's per-resource events do not account for every removal in its summary (an older `tofu`, a truncated stream), the gate counts every removal. -*On forge-cli `main`, not yet in a release ([forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007)).* On AWS the list is by type and, for what a forge Spark build ran with, by the end of the resource's name too. Its artifacts, role, log group and the key that encrypts the log groups, jobs and the product's own EMR Serverless application hold no data, so removing a Spark build is not a data loss. `apply` lists them (` resource(s) a removed build ran with (its artifacts, role, log group and job; no table data lost; not gated; its logs go with them)`) and passes without the flag. An output table, a bucket and a bucket's own key still count, and so does a stream's dead-letter table (`sink: table`), which goes with the build: removing that build needs the flag. See [When a build leaves the contract](../advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract). +*(since 0.34.0, [forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007))* On AWS the list is by type and, for what a forge Spark build ran with, by the end of the resource's name too. Its artifacts, role, log group and the key that encrypts the log groups, jobs and the product's own EMR Serverless application hold no data, so removing a Spark build is not a data loss. `apply` lists them (` resource(s) a removed build ran with (its artifacts, role, log group and job; no table data lost; not gated; its logs go with them)`) and passes without the flag. An output table, a bucket and a bucket's own key still count, and so does a stream's dead-letter table (`sink: table`), which goes with the build: removing that build needs the flag. See [When a build leaves the contract](../advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract). Some governance changes cannot be made to a live BigQuery table in place and plan its replacement, which this gate refuses without the flag: adding retention (`expire: true`) to a table that exists, adding or changing its Cloud KMS key, and removing `expire`. Roll these out as: plan, review the replacement, apply with `--allow-data-loss`, then run the build again to land the data. See [GCP](../providers/gcp.md). @@ -784,7 +784,7 @@ Options marked *native* are read only by the native engine. As of 0.18.1 the Ope | Option | Default | Description | | --- | --- | --- | -| `--mode` | `amend` | One of `dry-run`, `create-only`, `amend`, `amend-and-build`, `replace`, `replace-and-build`. *On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* Also `reconcile`: see [Reconcile mode](#reconcile-mode). | +| `--mode` | `amend` | One of `dry-run`, `create-only`, `amend`, `amend-and-build`, `replace`, `replace-and-build`. *(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* Also `reconcile`: see [Reconcile mode](#reconcile-mode). | | `--yes` | off | Skip confirmation. | | `--dry-run` | off | Alias for `--mode dry-run`. | | `--ensure-opentofu` | off | *(since 0.8.8)* If the `tofu` binary is missing, provision a **pinned, SHA-256-verified** OpenTofu build before a cloud apply — **no root, gpg, cosign, curl, or unzip needed** (Python stdlib only). Idempotent (a usable `tofu` at/above the engine floor is left untouched) and a no-op for native / `local` applies. Pin via `FLUID_OPENTOFU_VERSION`. `fluid generate ci` bakes this into the generated apply stage so cloud applies work on locked-down / non-root runners. | diff --git a/docs/cli/generate.md b/docs/cli/generate.md index 35bfcca..045e3c2 100644 --- a/docs/cli/generate.md +++ b/docs/cli/generate.md @@ -239,11 +239,11 @@ Model `access` is written under `config:` whatever version is detected. If no dbt is detectable, the generator emits the older shapes. dbt v2 rejects those, so a CI job that generates without dbt installed and then runs dbt v2 fails at parse. Install the dbt you will run in the job that runs `generate`, or pass `--dbt-tests-key` explicitly for the tests key. forge-cli's source records the floors above as measured on dbt-core 1.8.10, 1.9.11, 1.10.0, 1.10.5, 1.10.8, 1.12.5 and 2.0.4. -Up to 0.33.0, when `fluid apply` runs a dbt build in a container, the fallback when the local dbt cannot run the build's adapter, the container installs `dbt-<2` and `dbt-core<2` into `python:3.12-slim`. Three variables change that: `DBT_BOOTSTRAP_IMAGE` replaces the base image, `DBT_ADAPTER_PACKAGE` replaces the install (and drops the `dbt-core<2` cap, which is how you opt into v2), and `DBT_DOCKER_IMAGE` runs a prebuilt image that already has dbt. *On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* The fallback installs pinned packages into a pinned image, and the variables take pinned values only: see [Which dbt `fluid apply` runs](#which-dbt-fluid-apply-runs). +Up to 0.33.0, when `fluid apply` runs a dbt build in a container, the fallback when the local dbt cannot run the build's adapter, the container installs `dbt-<2` and `dbt-core<2` into `python:3.12-slim`. Three variables change that: `DBT_BOOTSTRAP_IMAGE` replaces the base image, `DBT_ADAPTER_PACKAGE` replaces the install (and drops the `dbt-core<2` cap, which is how you opt into v2), and `DBT_DOCKER_IMAGE` runs a prebuilt image that already has dbt. *(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* The fallback installs pinned packages into a pinned image, and the variables take pinned values only: see [Which dbt `fluid apply` runs](#which-dbt-fluid-apply-runs). #### Which dbt `fluid apply` runs -*On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* `fluid apply` runs a dbt build with the dbt that `$DBT_EXECUTABLE` names, as it is and whatever adapter it has. Without `$DBT_EXECUTABLE`, it uses `dbt` on `PATH`, or the `dbt` next to the Python that runs forge, when that dbt has the build's adapter. Otherwise, with Docker available, it runs the build in a container. +*(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* `fluid apply` runs a dbt build with the dbt that `$DBT_EXECUTABLE` names, as it is and whatever adapter it has. Without `$DBT_EXECUTABLE`, it uses `dbt` on `PATH`, or the `dbt` next to the Python that runs forge, when that dbt has the build's adapter. Otherwise, with Docker available, it runs the build in a container. **The adapter.** The build's `engine` names it, as `dbt-snowflake` names `snowflake`. The engines `dbt` and `dbt-core` name none, so the build's `execution.runtime.platform` does: `snowflake`, `bigquery` (and `gcp`), `duckdb`, `redshift` (and `aws`), `postgres` (and `postgresql`), `spark`, `athena` (and `aws-athena`), `glue` (and `aws-glue`) and `databricks`. dbt lists an adapter under its plugin name, which can differ from its package's: `dbt-athena-community` installs the plugin `athena`. For dbt-core v1, forge matches the plugin name exactly against the names listed under `Plugins:` in `dbt --version`. A dbt Fusion (v2), which lists none, is used for any adapter. In 0.33.0 the engine `dbt-athena-community` was looked for as `athena-community`, so a dbt that had the Athena plugin, found without `$DBT_EXECUTABLE`, was never used, and the build went to a container. A build that names no adapter runs on the local dbt as it is. diff --git a/docs/providers/aws.md b/docs/providers/aws.md index 577690a..b0c2c4b 100644 --- a/docs/providers/aws.md +++ b/docs/providers/aws.md @@ -359,7 +359,7 @@ The `aws_glue_catalog_table` now carries `open_table_format_input` with `iceberg An existing table is never replaced. Its state holds no `open_table_format_input`, the ignore keeps that value, and the plan shows no change. An engine's commits change `metadata_location` in `parameters`, which is ignored too. -*On forge-cli `main`, not yet in a release ([forge-cli #1012](https://github.com/Agenticstiger/forge-cli/pull/1012)).* The table's `location` is emitted as Iceberg keeps it: without a trailing slash, as `s3:///`, and as `s3://` for a table at a bucket's root. Iceberg strips the trailing slash from a table's location and writes the location back into the Glue table with every commit. With the slash that forge emitted before, each plan after a commit planned an update of the table, one for each table an engine had written to, and `fluid apply` sent an `UpdateTable` beside the commits of a stream that kept running. This is the plan that follows an Iceberg-style commit, run as forge-cli's own test of it plays it (the real OpenTofu against a local mock of AWS): the first line on the code before this change, the second after it: +*(since 0.34.0, [forge-cli #1012](https://github.com/Agenticstiger/forge-cli/pull/1012))* The table's `location` is emitted as Iceberg keeps it: without a trailing slash, as `s3:///`, and as `s3://` for a table at a bucket's root. Iceberg strips the trailing slash from a table's location and writes the location back into the Glue table with every commit. With the slash that forge emitted before, each plan after a commit planned an update of the table, one for each table an engine had written to, and `fluid apply` sent an `UpdateTable` beside the commits of a stream that kept running. This is the plan that follows an Iceberg-style commit, run as forge-cli's own test of it plays it (the real OpenTofu against a local mock of AWS): the first line on the code before this change, the second after it: ```text tofu plan: +0 ~1 -0 diff --git a/docs/upgrading.md b/docs/upgrading.md index 9392799..7e81752 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1451,50 +1451,50 @@ These changes are in 0.33.0. Several change what a plan or a generated file does - **New, opt-in: `fluid ship --journal-dir` keeps a stage journal, and every `--release-dir` run keeps one.** Each stage writes a `started` event before it runs and a `finished` event after, with its status and the digests of what it wrote. Before an `apply` that is not a dry run, an enforcing `policy-apply`, `publish` or a `schedule-sync` that is not a dry run, ship refuses the stage when an earlier attempt of it, or of `apply`, has no known outcome, until you inspect the target and pass `--acknowledge-inspected STAGE`. An `apply` that did not pass is refused until stage 6 has planned again. `--explain` shows what the journal would refuse. Without either flag no journal is kept and no stage is refused, and the run report records a stage killed by a signal (exit 129 to 159) as `unknown` instead of `failed`. See [The stage journal](./cli/ship.md#the-stage-journal). -## Not in a release yet +## Changes in 0.34.0 -These changes are on forge-cli `main`. The pin on this site is 0.33.0, so they are not in the release these pages document. +These changes are in 0.34.0. The pin on this site stays at 0.33.0 until the pages are checked against that release. -*On forge-cli `main`, not yet in a release ([forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008)).* +*(since 0.34.0, [forge-cli #1008](https://github.com/Agenticstiger/forge-cli/pull/1008))* - **`fluid apply` stops a removed Spark build's runs on AWS before it destroys what they run with.** When a plan deletes a build's resources, because the build left the contract or the product went, the apply cancels the build's active runs and waits for them (`FLUID_SPARK_WAIT_SECONDS`, 900 seconds by default), and empties and stops the product's own EMR Serverless application. It refuses first, with the `aws iam detach-*-policy` commands, when an IAM policy that the plan destroys is still attached, and a run that is still going after the wait refuses the apply before anything is destroyed. `--dry-run` says what it would cancel. The first apply after the upgrade adds three tags, `fluid_runtime`, `fluid_run_target` and `fluid_run_region`, in place to each Spark build's log group and to an EMR on EC2 build's job policy. The identity that runs `fluid apply` needs `iam:ListEntitiesForPolicy`, the list, get and cancel actions of the runtime that a removed build ran on, and, for the product's own application, `emr-serverless:GetApplication` and `emr-serverless:StopApplication` besides the EMR Serverless list, get and cancel actions. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract). -*On forge-cli `main`, not yet in a release ([forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007)).* +*(since 0.34.0, [forge-cli #1007](https://github.com/Agenticstiger/forge-cli/pull/1007))* - **Removing a Spark build on AWS needs no `--allow-data-loss`, unless the build has a dead-letter table.** The data-loss gate counted what a removed build ran with (its runner and job config objects, role and policies, log group and the key its logs are encrypted with, Glue job and connection, and the product's own EMR Serverless application) as data, so a plan that drops a build was refused unless the apply was allowed to destroy data, and that allowance also let a real data loss in the same plan through. They are now listed as removed and not gated. An output table, a bucket and a bucket's own key still count, and so does a stream's dead-letter table (`sink: table`), which goes with the build, so a plan that removes one of them needs the flag as before. A pipeline that passes it only to remove a build that has no dead-letter table no longer needs to. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract). -*On forge-cli `main`, not yet in a release ([forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015)).* +*(since 0.34.0, [forge-cli #1015](https://github.com/Agenticstiger/forge-cli/pull/1015))* - **A Spark stream can bound what one batch reads: `properties.stream.maxOffsetsPerTrigger` (0.7.7 preview).** A whole number from 1 to 9223372036854775807, passed to Spark's Kafka source, so a stream that catches up after a stop, or starts from `earliest`, lands in batches of bounded size, `trigger.availableNow` runs included. Unset, a batch reads everything that is waiting and the job config has no key. Managed Flink refuses the key. Setting it changes the job config, so a running stream restarts once on its next apply. See [Bounding a batch](./advanced/spark-iceberg-aws.md#bounding-a-batch). - **Dead letters sent to a topic are keyed by where they came from.** The key was the record's own key. It is `-@` now, and the record's own key is in the header `fluid.dead-letter.raw-key`, null for a record without a key and for a build that masks fields. A consumer that reads the key reads the header instead. A batch sent again after a failure carries the same keys, so a consumer drops repeats by key. Table dead letters are unchanged. See [The topic](./advanced/spark-iceberg-aws.md#the-topic). - **The runner sets `failOnDataLoss` to `true` on every Kafka source.** That is Spark's default, so a stream whose offsets the topic no longer holds fails, as before, and does not skip records. - **Each running Spark stream restarts once on its next apply, on its checkpoint.** The runner changed, and its sha256 is part of the config hash, so the hash of a build whose contract did not change is another one. -*On forge-cli `main`, not yet in a release ([forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014)).* +*(since 0.34.0, [forge-cli #1014](https://github.com/Agenticstiger/forge-cli/pull/1014))* - **A Spark run starts only on the job config that `fluid apply` compiled.** Every runtime, local included, passes the config's digest at submit (`--fluid-job-sha256`), and forge's runner refuses any other config before it reads anything. On AWS, `fluid apply` first reads the job config and the runner back from S3, and starts nothing unless they are the ones this apply compiled: the build fails with `or its runner is not the one this apply compiled`, and `fluid apply` ends with exit code `1`. A running job of another config is already cancelled by then, so run `fluid apply` again: it uploads both and starts the run. The identity that runs `fluid apply` needs `s3:GetObject` on both objects (the scheduler's `fluid--spark-caller` policy already holds `s3:GetObject`, but not `kms:Decrypt`). Write access to the artifact bucket stays the boundary. The runner changed, so each running Spark stream restarts once on its next apply, on its checkpoint. See [A run starts only on the config this apply compiled](./advanced/spark-iceberg-aws.md#a-run-starts-only-on-the-config-this-apply-compiled). -*On forge-cli `main`, not yet in a release ([forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009)).* +*(since 0.34.0, [forge-cli #1009](https://github.com/Agenticstiger/forge-cli/pull/1009))* - **`fluid apply` runs a dbt or Python build whatever its trigger.** A dbt or Python build of type `dataset`, `schedule_and_dataset`, `event`, `dependency` or `timetable` failed under `fluid apply` with `Unknown trigger type`. That included the task of the DAG that `fluid generate schedule` renders for a `dataset` or `schedule_and_dataset` build (`fluid apply --build-id`). The dbt and Python runners now run such a build once. `manual` keeps its iterations, `continuous` is still refused, and a `trigger.type` that is none of the names the schema knows is refused. See [What the DAG does](./advanced/airflow.md#what-the-dag-does). - **`fluid apply` finds the dbt that has the build's adapter.** An engine named after its package, `dbt-athena-community`, was looked for as `athena-community`, so a local dbt with the Athena plugin was never used. forge now asks for dbt's plugin name (`athena`), reads the adapter of an engine without one (`dbt`, `dbt-core`) from the build's platform, and matches the `Plugins:` list of `dbt --version` exactly. - **The dbt container fallback is pinned, and refuses what it cannot pin or authenticate.** It runs `python:3.12-slim` pinned by digest and installs `dbt-core==1.11.17` and an exact adapter package from wheels only, with pip run isolated against PyPI. `DBT_ADAPTER_PACKAGE` must list exact pins, `DBT_DOCKER_IMAGE` and `DBT_BOOTSTRAP_IMAGE` must be pinned by digest, and an adapter with no pin of its own (`glue`, for one) runs in a container only with `DBT_ADAPTER_PACKAGE` or `DBT_DOCKER_IMAGE`. A build of an AWS adapter (`athena`, `glue`, `redshift`) whose dbt profile names an AWS profile, or of `athena`, `glue` or a Redshift IAM build that has no AWS keys in the environment or the dbt profile, is refused before `docker` starts, with the command that installs the adapter next to dbt or exports short-lived keys. forge mounts nothing of `~/.aws`. See [Which dbt `fluid apply` runs](./cli/generate.md#which-dbt-fluid-apply-runs). -*On forge-cli `main`, not yet in a release ([forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010)).* +*(since 0.34.0, [forge-cli #1010](https://github.com/Agenticstiger/forge-cli/pull/1010))* - **A Spark job on AWS can write its own table where Lake Formation governs new tables.** forge granted a job's role Lake Formation access to its table only when the output's binding declared a non-empty `governance.lakeFormation` block, so in an account that removed the `IAM_ALLOWED_PRINCIPALS` default for new tables, a stream without that block could not see its own table. The grant (`INSERT`, `ALTER`, `DESCRIBE`, `SELECT`, and `DELETE` for a batch job) is now emitted for such an output and for a stream's dead-letter table beside it, with a `count` that the plan decides from the account's settings: 1 where Lake Formation governs new tables, 0 where IAM alone decides, so an account that keeps the default creates nothing new. The identity that runs `fluid plan` and `fluid apply` needs `lakeformation:GetDataLakeSettings`. An output with a non-empty `governance.lakeFormation` block keeps its grants, and an EMR on EC2 job, which runs as the cluster's own role, gets none. See [Lake Formation writes](./advanced/spark-iceberg-aws.md#lake-formation-writes). -*On forge-cli `main`, not yet in a release ([forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013)).* +*(since 0.34.0, [forge-cli #1013](https://github.com/Agenticstiger/forge-cli/pull/1013))* - **`fluid apply --mode reconcile` keeps the Spark streams running and deploys nothing.** It runs no `tofu`, no DDL and no build, and it takes no plan. It starts a stream that died again on its checkpoint, within its restart budget, when what the last apply deployed is what the contract compiles to. When it is not, it starts nothing, exits `1` and names what to run. `--build-id ` picks one stream, and a keepalive, a scheduler task every few minutes, is what runs it. See [Reconcile mode](./cli/apply.md#reconcile-mode). - **A Spark stream that died starts again within a restart budget, on a plain `fluid apply` too.** On EMR on EC2, Glue and a local build, a stream whose last run ended `FAILED` was an error until the build changed. It is started again on its checkpoint now, up to `execution.retries.maxAttempts` times per rolling hour (5 when unset, at most 10), each restart carrying `--fluid-restart-of `, and a spent budget is the error. EMR Serverless and EMR on EKS keep their own retries, and forge starts nothing on top of them. A run cancelled through the runtime is left stopped by `--mode reconcile` and started again by a plain `fluid apply`. `execution.retries` is no longer part of the config hash on EMR on EC2 and Glue, so a stream there that declares it restarts once on its next apply. A local run whose supervisor got a signal from outside forge, a shutdown of the machine say, reads as failed, not cancelled. Of two copies of one config the older is kept, where the newer was. See [The restart budget](./advanced/spark-iceberg-aws.md#the-restart-budget). - **The scheduler policy `fluid--spark-caller` covers always-on streams, and its statements change shape.** It lets a role start, watch and stop a stream's runs and read its job config and runner, so a keepalive can run on MWAA. It holds one statement for each runtime, with every build's target, role and artifacts, where it held one for each build, which reached the 6,144-character limit of a managed policy at about six builds. It holds no `kms:Decrypt`: a bucket that encrypts the artifacts with a KMS key needs it for the scheduler's role. See [The caller's permissions](./advanced/spark-iceberg-aws.md#the-caller-s-permissions). -*On forge-cli `main`, not yet in a release ([forge-cli #1017](https://github.com/Agenticstiger/forge-cli/pull/1017)).* +*(since 0.34.0, [forge-cli #1017](https://github.com/Agenticstiger/forge-cli/pull/1017))* - **A destroy waits for an EMR Serverless application that is still being created or started.** A `CREATING` or `STARTING` application is waited for until it settles: a created one is deleted as it is, a started one is stopped first, and one still moving after `FLUID_SPARK_WAIT_SECONDS` refuses the apply. The refusal's hint names `iam:ListEntitiesForPolicy`, `emr-serverless:GetApplication` and `emr-serverless:StopApplication`, which the identity that runs the destroy needs beyond the runtime's list and cancel actions. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract). -*On forge-cli `main`, not yet in a release ([forge-cli #1012](https://github.com/Agenticstiger/forge-cli/pull/1012)).* +*(since 0.34.0, [forge-cli #1012](https://github.com/Agenticstiger/forge-cli/pull/1012))* - **An Iceberg table on Glue plans no change after an engine commits to it.** forge emitted the table's location with a trailing slash. Iceberg keeps it without one and writes it back into Glue on every commit, so each `fluid apply` after a commit planned an update (`~1`) for each table that had been written, and sent an `UpdateTable` while a stream kept committing. The location is now emitted as Iceberg keeps it. The first apply after the upgrade updates once, in place, the location of each Iceberg table that Glue still holds with the trailing slash, and a table in a Hive format keeps its trailing slash. See [Iceberg tables in Glue](./providers/aws.md#iceberg-tables-in-glue). diff --git a/scripts/forge-cli-releases.json b/scripts/forge-cli-releases.json index 5da4ef8..2767aac 100644 --- a/scripts/forge-cli-releases.json +++ b/scripts/forge-cli-releases.json @@ -21,7 +21,7 @@ ], "repo": "Agenticstiger/forge-cli", "changelogUrl": "https://raw.githubusercontent.com/Agenticstiger/forge-cli/main/CHANGELOG.md", - "reviewedThroughRelease": "0.33.0", + "reviewedThroughRelease": "0.34.0", "releaseOfPr": { "707": "0.20.0", "709": "0.20.0", @@ -77,6 +77,15 @@ "1001": "0.33.0", "1002": "0.33.0", "1004": "0.33.0", - "1005": "0.33.0" + "1005": "0.33.0", + "1007": "0.34.0", + "1008": "0.34.0", + "1009": "0.34.0", + "1010": "0.34.0", + "1012": "0.34.0", + "1013": "0.34.0", + "1014": "0.34.0", + "1015": "0.34.0", + "1017": "0.34.0" } }