Skip to content

fix(iac): emit an Iceberg table's Glue location as Iceberg keeps it - #1012

Merged
fas89 merged 7 commits into
mainfrom
claude/glue-iceberg-location-drift-d88a8f
Nov 19, 2026
Merged

fas89 merged 7 commits into
mainfrom
claude/glue-iceberg-location-drift-d88a8f

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Every fluid apply after an engine committed to an Iceberg table on Glue planned ~1 per written
table (seen live: +0 ~1 -0 and +0 ~2 -0 on each scheduled re-apply beside a stream). forge
emitted the table's storage_descriptor.location with a trailing slash. Iceberg keeps a location
without one (LocationUtil.stripTrailingSlash) and writes it back into Glue on every commit
(GlueTableOperations.doCommit). The plan then sent an UpdateTable to put the slash back while
the stream kept committing:

  • the provider re-reads the live metadata_location right before the call
    (internal/service/glue/catalog_table.go), so a commit is lost only if it lands in that
    millisecond window;
  • but it sends previous_metadata_location and the other parameters from the plan's refresh,
    minutes old.

An Iceberg table's Glue location is now the warehouse location without its trailing slash, in the
OpenTofu emit and in the native planner's glue.ensure_iceberg_table action alike
(providers/aws/util/warehouse.py::iceberg_table_location), so the two keep deriving the same
string. A commit then leaves nothing to update. Hive-format tables keep the slash.

The first apply after the upgrade updates each existing Iceberg table's location once, in place
(the same update every apply made before). fluid verify compares locations normalized, so it
is unaffected.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Documentation

  • Docs PR linked
  • No docs needed — this change does not affect user-facing behaviour
    • Justification: the emitted location's form only; no contract field or command changes.
  • Docs TODO

Testing

  • tests/iac/test_iac_aws_iceberg_location_moto.py, a real tofu against moto (added to the
    Glue partition moto lane in iac-tests.yml): apply an Iceberg table, play a commit as
    GlueTableOperations makes it (location without the slash, new metadata_location), and plan.
    With this change: +0 ~0 -0. On the parent emit (scratch run): +0 ~1 -0.
  • tests/providers/aws/test_warehouse_derivation.py keeps the planner and the emit byte-identical;
    the Iceberg golden and the explicit-bucket pin say the slashless form.
  • tests/iac/test_iac_aws_iceberg_metadata.py: a table from a recorded earlier state is now
    updated once, in place (its location loses the slash), and still never replaced; every other
    resource is a no-op.
  • providers/aws and iac unit suites: 4,578 passed. A wider run (iac, providers, build_runners,
    cli, e2e) gave 14,279 passed. Its 8 failures are the local baseline (lazy-import budgets in the
    shared venv, two duckdb sandbox roots, two secret-ref worker cases, a Databricks SDK message
    under COLUMNS=200, a workspace-file case). None of them touches this change.
  • Gates: ruff, the isolated dead-code gate, black 24.10.0, license headers, lint-imports, the
    product guardrail, detect-secrets on the changed files, mypy strict hotspots.

forge emitted an Iceberg table's location with a trailing slash; Iceberg
keeps it without one (LocationUtil.stripTrailingSlash) and writes it back
into Glue on every commit (GlueTableOperations). So each fluid apply
after a commit planned ~1 per written table and sent an UpdateTable while
a stream kept committing, with previous_metadata_location from the plan's
refresh.

The Glue table location of an Iceberg table, in the OpenTofu emit and in
the native planner's glue.ensure_iceberg_table action alike, is now the
warehouse location without its trailing slash
(warehouse.iceberg_table_location). Hive-format tables keep the slash.

A real tofu run against moto plays a commit as GlueTableOperations makes
it, then plans: +0 ~0 -0 (the parent emit planned +0 ~1 -0).
@github-actions github-actions Bot added provider Changes or requests related to providers ci Continuous integration and automation changes security Security-related changes or reports tests Test coverage or test infrastructure changes labels Nov 19, 2026
The recorded state's table keeps the trailing slash forge emitted then,
so the plan now updates its location once, in place, and replaces
nothing: what the test is for. It asserts that the location is the only
change and that every other resource is a no-op.
…lue-iceberg-location-drift-d88a8f

# Conflicts:
#	CHANGELOG.md
…ither

From review: an Iceberg output with location.path "" still emitted
s3://bucket/, so Iceberg's s3://bucket put the plan back to ~1. The
location is built without the separator then, in the emit and (through
iceberg_table_location) the planner alike, and a test pins both.

The dead-letter test now also checks "beside, never inside" by whole
path segments: order_lines__dead_letter starts with order_lines as a
string, not as a directory. The dead letters' S3 prefix comes from the
contract's path, not from this location, and keeps its slash.
…lue-iceberg-location-drift-d88a8f

# Conflicts:
#	.secrets.baseline
@fas89
fas89 merged commit 92b7c8e into main Nov 19, 2026
69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Continuous integration and automation changes provider Changes or requests related to providers security Security-related changes or reports tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant