Repository navigation
fix(iac): emit an Iceberg table's Glue location as Iceberg keeps it - #1012
Merged
Merged
Conversation
forge emitted an Iceberg table's location with a trailing slash; Iceberg keeps it without one (LocationUtil.stripTrailingSlash) and writes it back into Glue on every commit (GlueTableOperations). So each fluid apply after a commit planned ~1 per written table and sent an UpdateTable while a stream kept committing, with previous_metadata_location from the plan's refresh. The Glue table location of an Iceberg table, in the OpenTofu emit and in the native planner's glue.ensure_iceberg_table action alike, is now the warehouse location without its trailing slash (warehouse.iceberg_table_location). Hive-format tables keep the slash. A real tofu run against moto plays a commit as GlueTableOperations makes it, then plans: +0 ~0 -0 (the parent emit planned +0 ~1 -0).
…lue-iceberg-location-drift-d88a8f
The recorded state's table keeps the trailing slash forge emitted then, so the plan now updates its location once, in place, and replaces nothing: what the test is for. It asserts that the location is the only change and that every other resource is a no-op.
…lue-iceberg-location-drift-d88a8f # Conflicts: # CHANGELOG.md
…ither From review: an Iceberg output with location.path "" still emitted s3://bucket/, so Iceberg's s3://bucket put the plan back to ~1. The location is built without the separator then, in the emit and (through iceberg_table_location) the planner alike, and a test pins both. The dead-letter test now also checks "beside, never inside" by whole path segments: order_lines__dead_letter starts with order_lines as a string, not as a directory. The dead letters' S3 prefix comes from the contract's path, not from this location, and keeps its slash.
…lue-iceberg-location-drift-d88a8f # Conflicts: # .secrets.baseline
…lue-iceberg-location-drift-d88a8f
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Every
fluid applyafter an engine committed to an Iceberg table on Glue planned~1per writtentable (seen live:
+0 ~1 -0and+0 ~2 -0on each scheduled re-apply beside a stream). forgeemitted the table's
storage_descriptor.locationwith a trailing slash. Iceberg keeps a locationwithout one (
LocationUtil.stripTrailingSlash) and writes it back into Glue on every commit(
GlueTableOperations.doCommit). The plan then sent anUpdateTableto put the slash back whilethe stream kept committing:
metadata_locationright before the call(
internal/service/glue/catalog_table.go), so a commit is lost only if it lands in thatmillisecond window;
previous_metadata_locationand the other parameters from the plan's refresh,minutes old.
An Iceberg table's Glue location is now the warehouse location without its trailing slash, in the
OpenTofu emit and in the native planner's
glue.ensure_iceberg_tableaction alike(
providers/aws/util/warehouse.py::iceberg_table_location), so the two keep deriving the samestring. A commit then leaves nothing to update. Hive-format tables keep the slash.
The first apply after the upgrade updates each existing Iceberg table's location once, in place
(the same update every apply made before).
fluid verifycompares locations normalized, so itis unaffected.
Type of Change
Documentation
Testing
tests/iac/test_iac_aws_iceberg_location_moto.py, a realtofuagainst moto (added to theGlue partition moto lane in iac-tests.yml): apply an Iceberg table, play a commit as
GlueTableOperationsmakes it (location without the slash, newmetadata_location), and plan.With this change:
+0 ~0 -0. On the parent emit (scratch run):+0 ~1 -0.tests/providers/aws/test_warehouse_derivation.pykeeps the planner and the emit byte-identical;the Iceberg golden and the explicit-bucket pin say the slashless form.
tests/iac/test_iac_aws_iceberg_metadata.py: a table from a recorded earlier state is nowupdated once, in place (its location loses the slash), and still never replaced; every other
resource is a no-op.
cli, e2e) gave 14,279 passed. Its 8 failures are the local baseline (lazy-import budgets in the
shared venv, two duckdb sandbox roots, two secret-ref worker cases, a Databricks SDK message
under
COLUMNS=200, a workspace-file case). None of them touches this change.product guardrail, detect-secrets on the changed files, mypy strict hotspots.