Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/providers/aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,15 @@ The `aws_glue_catalog_table` now carries `open_table_format_input` with `iceberg

An existing table is never replaced. Its state holds no `open_table_format_input`, the ignore keeps that value, and the plan shows no change. An engine's commits change `metadata_location` in `parameters`, which is ignored too.

*On forge-cli `main`, not yet in a release ([forge-cli #1012](https://github.com/Agenticstiger/forge-cli/pull/1012)).* The table's `location` is emitted as Iceberg keeps it: without a trailing slash, as `s3://<bucket>/<path>`, and as `s3://<bucket>` for a table at a bucket's root. Iceberg strips the trailing slash from a table's location and writes the location back into the Glue table with every commit. With the slash that forge emitted before, each plan after a commit planned an update of the table, one for each table an engine had written to, and `fluid apply` sent an `UpdateTable` beside the commits of a stream that kept running. This is the plan that follows an Iceberg-style commit, run as forge-cli's own test of it plays it (the real OpenTofu against a local mock of AWS): the first line on the code before this change, the second after it:

```text
tofu plan: +0 ~1 -0
tofu plan: +0 ~0 -0
```

The first `fluid apply` after the upgrade updates once, in place, the location of each Iceberg table that Glue still holds with the trailing slash. A table that an engine has committed to since the last apply has none already, and the plan shows no change for it. A table in a Hive format, such as Parquet, keeps its trailing slash.

**A table created before this change** has no metadata, so Athena still refuses it. Give it metadata only once you know it is empty:

1. Check that no engine has written to it. Its Glue parameters must have no `metadata_location`, and its prefix must hold no data files:
Expand Down
4 changes: 4 additions & 0 deletions docs/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -1494,6 +1494,10 @@ These changes are on forge-cli `main`. The pin on this site is 0.33.0, so they a

- **A destroy waits for an EMR Serverless application that is still being created or started.** A `CREATING` or `STARTING` application is waited for until it settles: a created one is deleted as it is, a started one is stopped first, and one still moving after `FLUID_SPARK_WAIT_SECONDS` refuses the apply. The refusal's hint names `iam:ListEntitiesForPolicy`, `emr-serverless:GetApplication` and `emr-serverless:StopApplication`, which the identity that runs the destroy needs beyond the runtime's list and cancel actions. See [When a build leaves the contract](./advanced/spark-iceberg-aws.md#when-a-build-leaves-the-contract).

*On forge-cli `main`, not yet in a release ([forge-cli #1012](https://github.com/Agenticstiger/forge-cli/pull/1012)).*

- **An Iceberg table on Glue plans no change after an engine commits to it.** forge emitted the table's location with a trailing slash. Iceberg keeps it without one and writes it back into Glue on every commit, so each `fluid apply` after a commit planned an update (`~1`) for each table that had been written, and sent an `UpdateTable` while a stream kept committing. The location is now emitted as Iceberg keeps it. The first apply after the upgrade updates once, in place, the location of each Iceberg table that Glue still holds with the trailing slash, and a table in a Hive format keeps its trailing slash. See [Iceberg tables in Glue](./providers/aws.md#iceberg-tables-in-glue).

## Older releases

Upgrading from before `0.15.0`? Start with the
Expand Down
Loading