Skip to content

fix: a plan made from a --release-dir release applies, also from another checkout - #1022

Merged
fas89 merged 3 commits into
mainfrom
fix/release-packet-provenance
Nov 19, 2026
Merged

fas89 merged 3 commits into
mainfrom
fix/release-packet-provenance

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

A CI release flow that publishes a packet with fluid ship --release-dir, plans the packet's bundle in a later job, and applies that plan after approval could not reach apply for a contract with a dbt build. Two causes, two fixes:

  1. The release had no provenance file. --release-dir copied bundle.tgz without bundle.tgz.provenance.json, so a plan made from the release's bundle recorded no source contract. fluid apply then refused with apply_plan_digest_bundle_mismatch ("the checkout to check it against cannot be found").
    • publish() now writes the file into the release, with the same contract relative to the release directory. It uses bundle._write_provenance, resolved as readers resolve a bundle's directory.
    • release.json lists it in files. identity is unchanged.
    • fluid ship warns (release_provenance_missing) when it cannot write one.
  2. A plan records the planning checkout's absolute paths. When a later job checked the repository out at another path, apply refused for the same reason.
    • The recorded paths, covered by planDigest, still come first.
    • When they are gone, _contract_loader._resolve_plan_source_contract resolves the contract through the provenance file beside the bundle that apply has just verified against the plan. A context variable carries that bundle, and _scopes_dbt_binding resets it per apply.
    • The file is outside every digest, so the contract it names must declare the bundled id, and the dbt files there are still compared with the packed copy.
    • --no-verify-plan-binding sets nothing.

Reuse. The provenance file is the existing one (Helm's .prov pattern, already in bundle.py). The per-run scope mirrors project_files.set_bound_projects. Outside search: Terraform embeds a config snapshot in its saved plans so apply never needs the planning path. GitLab Runner's default build path holds the runner and a concurrency slot, so two jobs of one pipeline can check out at different paths. Here the plan's digest-covered path stays first, and a path relative to the verified artifact is the fallback.

Validation

  • New tests:
    • the release names its source contract from where it sits;
    • the resolver order;
    • _verify_plan_digests notes the bundle, and the binding-path anchor, IaC source and Spark workspace follow it;
    • end to end: ship → plan from the release → apply --dry-run, then again in a moved checkout with the planning one gone, where an edited model is refused by name.
  • Both end-to-end tests failed before the change with apply_plan_digest_bundle_mismatch.
  • Ship and anchor suites: 287 passed.
  • Full suite: 41,608 passed. The 163 local failures are identical on origin/main d5ed0fb (venv and basetemp environment).
  • Gates: ruff, black 24.10.0, dead-code sweep, guardrail, license headers, import contracts and detect-secrets 1.5.0 all pass.
  • contextvars was already loaded on the fluid --help path: 944 modules before and after.
  • Live, with the real CLI: a plan made from a release packet applies a dbt build (it stops at a refused tofu init), and so does one applied from another checkout path.

ripwire --quality-delta. The complexity regression is fixed. Rows left on purpose:

  • the three-line contextvar scope idiom, matching other scopes;
  • a test fixture's dbt build dict, similar to test_verify_databricks.py::contract_with_job;
  • short-horizon churn on the two apply functions the fix has to touch.

Reviews

  • Security review: no HIGH or MEDIUM. A tampered provenance file reaches only what the old plan-dir fallback reached, and dbt code is still byte-checked.
  • Delivery review: three P2 gaps (silent skip at ship, tests beyond the dbt check, changelog wording), fixed in the last commit and re-checked.
  • Defect scan: no Critical, High or Medium.

fluid ship --release-dir copied bundle.tgz without the provenance file
fluid bundle writes beside it, so a plan made from the release's bundle
recorded no source contract, and fluid apply refused a contract with a dbt
build (apply_plan_digest_bundle_mismatch: the checkout to check it against
cannot be found).

publish() now writes bundle.tgz.provenance.json into the release with the
same source contract, relative to the release directory (resolved as its
readers resolve the bundle's directory), through bundle._write_provenance.
release.json lists it in files; identity is unchanged.

The file's format is bundle.py's, which borrows Helm's <chart>.tgz.prov
pattern: a file beside the archive that names the archive's sha256.
A plan records the planning checkout's absolute paths. When a later CI job
checks the repository out at another path, fluid apply plan.json --bundle B
could not find a dbt build's project and refused the plan.

When the recorded paths are gone, the plan now resolves its source contract
through the bundle apply has just verified it against (its provenance file,
relative to the bundle), for the dbt check, the build runners, binding paths
and the IaC source. The recorded paths, under planDigest, still come first.
The provenance file is unauthenticated, so resolve_source_contract's id check
applies and the dbt files are still compared with the packed copy. Scoped per
apply by a context variable, reset when the run returns.

Pattern: locate the tree relative to the verified artifact, as Helm's .prov
sidecar names files relative to its archive.
A bundle that names no usable source contract, or a release on another
drive, still publishes without bundle.tgz.provenance.json, and a later apply
of a plan made from it refuses a dbt build. fluid ship now says so where the
release is made (release_provenance_missing).

Tests now cover the apply side beyond the dbt check: _verify_plan_digests
notes the verified bundle, and the binding-path anchor, the IaC source and
the Spark workspace follow it while the run's scope holds. The changelog
drops an unsourced GitLab claim and says the release must keep its place in
the checkout.
@github-actions github-actions Bot added cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes needs-docs Pull request needs a linked docs update or justification labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

  • Link a docs PR — check the "Docs PR linked" box and paste the URL
  • Mark as no docs needed — check "No docs needed" with a justification
  • Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.

@fas89
fas89 enabled auto-merge (squash) November 19, 2026 10:36
@fas89
fas89 merged commit f7b3d03 into main Nov 19, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli Changes to the CLI surface or implementation needs-docs Pull request needs a linked docs update or justification tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant