From 2e843979b179c658cc61427b56c195189507cb60 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 09:20:41 +0100 Subject: [PATCH] docs(aws): an Iceberg table on Glue plans no change after an engine commits to it (forge-cli #1012) The location is emitted as Iceberg keeps it, without the trailing slash. --- docs/providers/aws.md | 9 +++++++++ docs/upgrading.md | 4 ++++ 2 files changed, 13 insertions(+) diff --git a/docs/providers/aws.md b/docs/providers/aws.md index 24ea807..577690a 100644 --- a/docs/providers/aws.md +++ b/docs/providers/aws.md @@ -359,6 +359,15 @@ The `aws_glue_catalog_table` now carries `open_table_format_input` with `iceberg An existing table is never replaced. Its state holds no `open_table_format_input`, the ignore keeps that value, and the plan shows no change. An engine's commits change `metadata_location` in `parameters`, which is ignored too. +*On forge-cli `main`, not yet in a release ([forge-cli #1012](https://github.com/Agenticstiger/forge-cli/pull/1012)).* The table's `location` is emitted as Iceberg keeps it: without a trailing slash, as `s3:///`, and as `s3://` for a table at a bucket's root. Iceberg strips the trailing slash from a table's location and writes the location back into the Glue table with every commit. With the slash that forge emitted before, each plan after a commit planned an update of the table, one for each table an engine had written to, and `fluid apply` sent an `UpdateTable` beside the commits of a stream that kept running. This is the plan that follows an Iceberg-style commit, run as forge-cli's own test of it plays it (the real OpenTofu against a local mock of AWS): the first line on the code before this change, the second after it: + +```text +tofu plan: +0 ~1 -0 +tofu plan: +0 ~0 -0 +``` + +The first `fluid apply` after the upgrade updates once, in place, the location of each Iceberg table that Glue still holds with the trailing slash. A table that an engine has committed to since the last apply has none already, and the plan shows no change for it. A table in a Hive format, such as Parquet, keeps its trailing slash. + **A table created before this change** has no metadata, so Athena still refuses it. Give it metadata only once you know it is empty: 1. Check that no engine has written to it. Its Glue parameters must have no `metadata_location`, and its prefix must hold no data files: diff --git a/docs/upgrading.md b/docs/upgrading.md index 60f8626..8293166 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1484,6 +1484,10 @@ These changes are on forge-cli `main`. The pin on this site is 0.33.0, so they a - **A Spark job on AWS can write its own table where Lake Formation governs new tables.** forge granted a job's role Lake Formation access to its table only when the output's binding declared a non-empty `governance.lakeFormation` block, so in an account that removed the `IAM_ALLOWED_PRINCIPALS` default for new tables, a stream without that block could not see its own table. The grant (`INSERT`, `ALTER`, `DESCRIBE`, `SELECT`, and `DELETE` for a batch job) is now emitted for such an output and for a stream's dead-letter table beside it, with a `count` that the plan decides from the account's settings: 1 where Lake Formation governs new tables, 0 where IAM alone decides, so an account that keeps the default creates nothing new. The identity that runs `fluid plan` and `fluid apply` needs `lakeformation:GetDataLakeSettings`. An output with a non-empty `governance.lakeFormation` block keeps its grants, and an EMR on EC2 job, which runs as the cluster's own role, gets none. See [Lake Formation writes](./advanced/spark-iceberg-aws.md#lake-formation-writes). +*On forge-cli `main`, not yet in a release ([forge-cli #1012](https://github.com/Agenticstiger/forge-cli/pull/1012)).* + +- **An Iceberg table on Glue plans no change after an engine commits to it.** forge emitted the table's location with a trailing slash. Iceberg keeps it without one and writes it back into Glue on every commit, so each `fluid apply` after a commit planned an update (`~1`) for each table that had been written, and sent an `UpdateTable` while a stream kept committing. The location is now emitted as Iceberg keeps it. The first apply after the upgrade updates once, in place, the location of each Iceberg table that Glue still holds with the trailing slash, and a table in a Hive format keeps its trailing slash. See [Iceberg tables in Glue](./providers/aws.md#iceberg-tables-in-glue). + ## Older releases Upgrading from before `0.15.0`? Start with the