From 2d0a135fff043b5bf93ad22995be4d49266f23fd Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 11:46:14 +0100 Subject: [PATCH 1/7] docs: forge-cli 0.35.0, the release packet keeps its provenance file (forge-cli #1022) --- docs/cli/apply.md | 2 +- docs/cli/bundle.md | 1 + docs/cli/ship.md | 3 ++- docs/upgrading.md | 9 +++++++++ 4 files changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/cli/apply.md b/docs/cli/apply.md index c07d6d2..568f257 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -376,7 +376,7 @@ What the check does not cover: - `--no-verify-plan-binding` skips it and logs a `WARNING`, as it skips the digests. - Installed packages are not bound. dbt reads whatever `packages-install-path` holds, and `fluid apply` neither installs nor checks it. Run `dbt deps` from `package-lock.yml` in the job that applies. - A plan made from a contract file binds no dbt project. A build whose `repository` is a Git URL is not packed, because its ref pins it. -- Plan and apply must run in the same checkout path. The plan records the absolute path of the source contract, and `fluid apply` compares the project at that path, whatever checkout you run it from. If the path is gone, it refuses with the same event and says the checkout cannot be found. *(since 0.33.0, [forge-cli #985](https://github.com/Agenticstiger/forge-cli/pull/985))* A bundle copied without its [provenance file](./bundle.md#the-provenance-file) is refused the same way. +- Plan and apply must run in the same checkout path. The plan records the absolute path of the source contract, and `fluid apply` compares the project at that path, whatever checkout you run it from. If the path is gone, it refuses with the same event and says the checkout cannot be found. *(since 0.33.0, [forge-cli #985](https://github.com/Agenticstiger/forge-cli/pull/985))* A bundle copied without its [provenance file](./bundle.md#the-provenance-file) is refused the same way. Through 0.34.0 a plan applied in another checkout path was refused. *(since 0.35.0, [forge-cli #1022](https://github.com/Agenticstiger/forge-cli/pull/1022))* When the paths the plan records are gone, `fluid apply` finds the contract through the provenance file beside the bundle it has just verified against the plan. The recorded paths, which `planDigest` covers, still come first. The provenance file is outside every digest, so the contract it names must declare the bundled contract's id, and the dbt files there must match the copy the bundle packs: an edited file is refused by name. With `--no-verify-plan-binding` nothing is verified, so there is no fallback. *([forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714), since 0.20.0)* After these gates the contract the plan embeds is validated against its schema, and a plan that embeds none is refused with `apply_plan_missing_contract`. See [Contract validation](#contract-validation). diff --git a/docs/cli/bundle.md b/docs/cli/bundle.md index 8da1032..674ac50 100644 --- a/docs/cli/bundle.md +++ b/docs/cli/bundle.md @@ -433,6 +433,7 @@ Without a usable file, `fluid plan` and `fluid apply` log `source_contract_unresolved` and anchor relative paths at the bundle's own directory. Copy the bundle together with its provenance file, to a place where the recorded relative path still reaches the contract, or expect that. +*(since 0.35.0, [forge-cli #1022](https://github.com/Agenticstiger/forge-cli/pull/1022))* `fluid ship --release-dir` writes one into the release, for the contract relative to the release directory. For a bundle that packs a dbt project it is stricter: the checkout to compare against cannot be found, and `fluid apply` refuses with `apply_plan_digest_bundle_mismatch`, saying the bundle does not record a source diff --git a/docs/cli/ship.md b/docs/cli/ship.md index d801fab..516d7c3 100644 --- a/docs/cli/ship.md +++ b/docs/cli/ship.md @@ -230,12 +230,13 @@ The run stages into a new `DIR.partial--*` beside `DIR`. It copies the b releases/orders-1/ ├── artifacts/ the files artifacts/MANIFEST.json lists, with the MANIFEST ├── bundle.tgz +├── bundle.tgz.provenance.json (since 0.35.0, when the bundle names a source contract) ├── plan.json ├── release.json └── reports/ ship-report.json, ship-junit.xml, and each stage's report ``` -A run that fails, or is interrupted, publishes nothing. Its reports stay in `runtime/` and in the staging directory, and `ship-report.json` names the stage that stopped it. `DIR` does not exist after a failed run. A release run first removes the stage reports in `runtime/`, so only this run's are published. The release directory holds the bundle but does not copy the bundle's `.provenance.json` file (see [The provenance file](./bundle.md#the-provenance-file)); the release's `identity` does not need it. A release run reads only local files: it opens no network connection of its own. +A run that fails, or is interrupted, publishes nothing. Its reports stay in `runtime/` and in the staging directory, and `ship-report.json` names the stage that stopped it. `DIR` does not exist after a failed run. A release run first removes the stage reports in `runtime/`, so only this run's are published. Through 0.34.0 the release directory held the bundle but not its `.provenance.json` file, so a plan made from the release's bundle recorded no source contract and `fluid apply` refused a contract with a dbt build. *(since 0.35.0, [forge-cli #1022](https://github.com/Agenticstiger/forge-cli/pull/1022))* The release holds `bundle.tgz.provenance.json`, written for the release's own place: the same source contract, relative to the release directory (see [The provenance file](./bundle.md#the-provenance-file)). `release.json` lists it in `files` with its sha256, and it stays out of `identity`. A bundle that names no source contract gets no file, and neither does a release on another drive. `fluid ship` then warns `release_provenance_missing`. The file names the contract relative to where the release was published, so keep the release at the same place in the checkout, as CI artifacts restored into the same layout are. A release run reads only local files: it opens no network connection of its own. **Rules.** `DIR` must not exist, and stages 1 to 4 must run in the invocation. Otherwise ship refuses with exit `2`, and `--explain` says why: diff --git a/docs/upgrading.md b/docs/upgrading.md index 550a8f9..837c0f9 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1502,6 +1502,15 @@ These changes are in 0.34.0. - **`fluid generate schedule` writes a keepalive DAG for an always-on stream that declares `execution.retries`.** On Airflow, a Spark stream that is not `trigger.availableNow` and not `trigger.type: continuous` gets a DAG whose one task, `fluid_keepalive`, runs `fluid apply --mode reconcile --build-id --yes` every 5 minutes, or on the build's `execution.orchestration.airflow.dagConfig.schedule`. A stream that died starts again on its checkpoint, within the restart budget that `execution.retries` sets, and a run fails when the stream cannot be kept running. A stream that already declares `execution.retries` gets the DAG from its next `fluid generate schedule` or `fluid generate artifacts`. A stream without `execution.retries` gets no DAG, as before. A contract whose only builds are such streams gets this DAG instead of `No build needs a DAG`; Prefect, Dagster and `fluid scaffold-composer` still write nothing for it. The worker needs the `FLUID_SPARK_*` settings and the forge release that the CI apply has. See [Keeping a stream running](./advanced/airflow.md#keeping-a-stream-running). +## Changes in 0.35.0 + +These changes are in 0.35.0. + +*(since 0.35.0, [forge-cli #1022](https://github.com/Agenticstiger/forge-cli/pull/1022))* + +- **A plan made from a `fluid ship --release-dir` release can be applied when the contract has a dbt build.** The release now holds `bundle.tgz.provenance.json`, listed in `release.json` `files` and outside `identity`. Through 0.34.0 `fluid apply` refused such a plan with `apply_plan_digest_bundle_mismatch`. `fluid ship` warns `release_provenance_missing` when it cannot write the file. See [Publishing a release](./cli/ship.md#publishing-a-release). +- **A plan can be applied in another checkout of the same tree.** When the paths the plan records are gone, `fluid apply` finds the contract through the provenance file beside the bundle it verified the plan against. The dbt files must still match the packed copy. See [The dbt project](./cli/apply.md#the-dbt-project). + ## Older releases Upgrading from before `0.15.0`? Start with the From 7d621e032f12c2a949a2f8cfee840ffeb3462d59 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:00:16 +0100 Subject: [PATCH 2/7] docs: apply opens the checkout-path note with its 0.34.0 history --- docs/cli/apply.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli/apply.md b/docs/cli/apply.md index 568f257..526ea63 100644 --- a/docs/cli/apply.md +++ b/docs/cli/apply.md @@ -376,7 +376,7 @@ What the check does not cover: - `--no-verify-plan-binding` skips it and logs a `WARNING`, as it skips the digests. - Installed packages are not bound. dbt reads whatever `packages-install-path` holds, and `fluid apply` neither installs nor checks it. Run `dbt deps` from `package-lock.yml` in the job that applies. - A plan made from a contract file binds no dbt project. A build whose `repository` is a Git URL is not packed, because its ref pins it. -- Plan and apply must run in the same checkout path. The plan records the absolute path of the source contract, and `fluid apply` compares the project at that path, whatever checkout you run it from. If the path is gone, it refuses with the same event and says the checkout cannot be found. *(since 0.33.0, [forge-cli #985](https://github.com/Agenticstiger/forge-cli/pull/985))* A bundle copied without its [provenance file](./bundle.md#the-provenance-file) is refused the same way. Through 0.34.0 a plan applied in another checkout path was refused. *(since 0.35.0, [forge-cli #1022](https://github.com/Agenticstiger/forge-cli/pull/1022))* When the paths the plan records are gone, `fluid apply` finds the contract through the provenance file beside the bundle it has just verified against the plan. The recorded paths, which `planDigest` covers, still come first. The provenance file is outside every digest, so the contract it names must declare the bundled contract's id, and the dbt files there must match the copy the bundle packs: an edited file is refused by name. With `--no-verify-plan-binding` nothing is verified, so there is no fallback. +- Through 0.34.0, plan and apply had to run in the same checkout path: a plan records the planning checkout's absolute paths. The plan records the absolute path of the source contract, and `fluid apply` compares the project at that path, whatever checkout you run it from. If the path is gone, it refused with the same event and said the checkout cannot be found. *(since 0.33.0, [forge-cli #985](https://github.com/Agenticstiger/forge-cli/pull/985))* A bundle copied without its [provenance file](./bundle.md#the-provenance-file) is refused the same way. Since 0.35.0, when those paths are gone, apply finds the contract through the provenance file beside the bundle it has just verified against the plan *(since 0.35.0, [forge-cli #1022](https://github.com/Agenticstiger/forge-cli/pull/1022))*. The recorded paths, which `planDigest` covers, still come first. The provenance file is outside every digest, so the contract it names must declare the bundled contract's id. The dbt files there must match the copy the bundle packs, and an edited file is refused by name. With `--no-verify-plan-binding` nothing is verified, so there is no fallback. *([forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714), since 0.20.0)* After these gates the contract the plan embeds is validated against its schema, and a plan that embeds none is refused with `apply_plan_missing_contract`. See [Contract validation](#contract-validation). From 71919b73725c8a75a33e4c4f10186289ac043c71 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:02:12 +0100 Subject: [PATCH 3/7] docs: fluid profile needs flux-stats 0.7.0, quantile bins and the joint check (forge-cli #1021) --- docs/cli/profile.md | 18 ++++++++++++------ docs/upgrading.md | 4 ++++ 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/cli/profile.md b/docs/cli/profile.md index 852333d..f71dad0 100644 --- a/docs/cli/profile.md +++ b/docs/cli/profile.md @@ -6,11 +6,11 @@ Profile one expose's records where they live, and write a profile document of ou FLUX's statistic registry, `flux-stats`, reads every value and forms every statistic. The fidelity check uses the same registry, so a profile and a check of the same records hold the same numbers. -The `[profile]` extra installs DuckDB and `flux-stats` (`>=0.6.0,<0.7`) from PyPI. Without flux-stats, `fluid profile` stops before it reads anything and says what to install: +The `[profile]` extra installs DuckDB and `flux-stats` from PyPI. 0.34.0 needed `>=0.6.2,<0.7`. *(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* It needs `flux-stats[yaml]>=0.7.0,<0.8`. Without flux-stats, `fluid profile` stops before it reads anything and says what to install: ```text ❌ profile_statistics_unavailable [ERR_PROFILE_STATISTICS_UNAVAILABLE] - detail: fluid profile forms every statistic with flux-stats, the statistic registry FLUX shares with its fidelity check, and flux-stats is not installed. It needs flux-stats>=0.6.0,<0.7; install it with: pip install 'data-product-forge[profile]' + detail: fluid profile forms every statistic with flux-stats, the statistic registry FLUX shares with its fidelity check, and flux-stats is not installed. It needs flux-stats[yaml]>=0.7.0,<0.8; install it with: pip install 'data-product-forge[profile]' read: nothing: no record was read next: pip install 'data-product-forge[profile]' ``` @@ -155,18 +155,22 @@ For each property, the document holds the statistic the engine compares it with: | `marginalHistogram` | `marginalHistogram/1`: a count for each of the property's bins | | `levelHistogram` | `levelHistogram/1`: a count for each level the column declares | | `support` | `support/1`: the declared levels the records hold | -| `quantile` | `quantile/1`: the value at the property's `level`, in steps of the grid, with the number of records below it and at or below it, both rounded like every count | +| `quantile` | `quantile/1`: the value at the property's `level`, in steps of the grid, with the number of records below it and at or below it, both rounded like every count. *(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* A `quantile` that declares `bins` is answered with `quantile/2` instead, see below | | `correlation` | nothing: see below | A property with a `groupBy` gets one group for each level of that column the records hold, and one without gets a single group. The grid is the block's `encoding.fractionalBits`, 16 when the block declares none, and a `--fractional-bits` that differs is refused with `profile_fractional_bits_conflict`. Each number column still gets the [ungrouped `mean/1`](#the-mean-1-statistic), written once when the block declares that same mean. The block is read the way the FLUX engine reads it, and the fidelity grammar in flux-stats parses it, so a profile and a check read the same declarations. Before any record is read, `fluid profile` refuses a block the engine would refuse (`profile_bundle_block_refused`, with the engine's code), and a property over a column the profile does not read, or reads in another domain than the property does (`profile_bundle_column_not_profiled`, `profile_bundle_column_domain`). A record value outside the block's declared range for its column is refused with `profile_bundle_record_refused`, and so is a level outside the declared support where a level histogram or a `groupBy` reads the column; nothing is written, and the refusal names the row, never the value. A `support` lists the levels the records hold, as the engine reads them. A bundle that gives no block is refused before any record is read, with `profile_bundle_not_a_directory`, `profile_bundle_no_simulation`, `profile_bundle_many_simulations`, `profile_bundle_no_block` or `profile_bundle_unreadable`. `profile_bundle_unreadable` also refuses a block that holds more than 100,000 values once its YAML aliases are written out, nests deeper than 64 levels, or holds a number of more than 1,000 digits written out, and a bundle whose YAML merge keys (`<<`) copy more than 100,000 pairs. -`--bundle` needs flux-stats 0.6.2 or newer, the first release that holds the fidelity grammar, and on `main` the `[profile]` extra asks for `>=0.6.2,<0.7`. With an older flux-stats, `--bundle` stops before it reads anything: +*(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* A `quantile` property that declares `bins` is answered with `quantile/2`. The group states the declared bin that holds the order statistic at its level, and the records below that bin and through it, and no witness. A reader checks that the rounded counts still put the order statistic in that bin, and a group whose rounded counts do not is withheld. A profile that holds one names format `1-1-0`, and every other profile keeps `1-0-0` and its bytes. A quantile with bins is published beside a histogram that shares its bins, where a `quantile/1` is withheld. + +*(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* `--bundle` reads the bundle's YAML with flux-stats' exact reader and puts its block in the canonical form with flux-stats, where 0.34.0 used a copy of the engine's code. The bounds hold as before and a refusal still names the file, and the line where the reader reports one, never a value. Its words differ in places: a YAML error names the line and column where the reader found it, the most values a document may write out through its aliases is 1,000,000, and the characters it writes out are bounded too. A block whose bins are not strictly increasing grid points is refused before any record is read (`FLX-5004`), as the fidelity check refuses it. + +`--bundle` needs flux-stats 0.6.2 or newer, the first release that holds the fidelity grammar, and through 0.34.0 the `[profile]` extra asked for `>=0.6.2,<0.7`. *(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* It asks for `flux-stats[yaml]>=0.7.0,<0.8`, the first release that holds the readers and the check below. With an older flux-stats, `--bundle` stops before it reads anything: ```text ❌ profile_statistics_unavailable [ERR_PROFILE_STATISTICS_UNAVAILABLE] - detail: --bundle reads the fidelity block with flux-stats, and the installed flux-stats lacks flux_stats.fidelity. It needs flux-stats>=0.6.2,<0.7 + detail: --bundle reads the fidelity block with flux-stats, and the installed flux-stats lacks flux_stats.fidelity. It needs flux-stats[yaml]>=0.7.0,<0.8 read: nothing: no record was read next: pip install 'data-product-forge[profile]' ``` @@ -196,7 +200,9 @@ What the checks leave readable, on purpose: a grouped statistic lists every grou *(since 0.33.0, [forge-cli #967](https://github.com/Agenticstiger/forge-cli/pull/967), [forge-cli #978](https://github.com/Agenticstiger/forge-cli/pull/978))* One thing changes in that paragraph: a histogram or quantile group withheld for its size now brings secondary suppression, as the checks above say, and its siblings are withheld too. A withheld histogram cell still does not. The checks also leave two more things readable, or open: - A published `min` shows that no record lies below it, and a published `max` that none lies above it. That is a range of no records, and no check withholds a `min` or a `max` for it. A quantile group whose value is the smallest or the largest of its records shows the same, and `acro-default` holds zeros disclosive, so the group is withheld for that. -- The checks do not close one gap yet. A reader can still subtract one histogram's published counts from another's, or from a level histogram's group size, and read a withheld cell, a zero included, to within the rounding. forge-cli plans to close it with FLUX's `check_edge_counts`, which a flux-stats release has to carry first, and to ship that in a later release. +- Through 0.34.0 the checks did not close one gap. A reader can still subtract one histogram's published counts from another's, or from a level histogram's group size, and read a withheld cell, a zero included, to within the rounding. forge-cli plans to close it with FLUX's `check_edge_counts`, which a flux-stats release has to carry first, and to ship that in a later release. + +*(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* Through 0.34.0 the last point above was an open gap. Now `fluid profile` passes every histogram, level histogram and quantile of a column to flux-stats' `check_edge_counts` in one call, and publishes what it lets through. A histogram and a coarser one of one column, or a published cell and the column's count, no longer give a withheld cell back by subtraction, and a cell that does goes with it. A level histogram is checked over its levels as it is over bins, so one small level withholds a neighbour. A histogram that does not sit on the column's one partition of bins is withheld, and so is a group whose rounded counts no longer prove its quantile. The summary counts a cell or group withheld for its size or rounding as `A`, and one withheld so that another cannot be recovered as `D`. The check does not see a column's `support/1`, the group sizes of a statistic grouped by the column, or a declared level that holds no record, which is absent from a grouped statistic. A `correlation` property is not answered: its moments need the exact number of records, which the checks round. The summary names it, and `fluxsim verify` fails with `FLX-5037`, so a block you check against a profile cannot declare a correlation yet. diff --git a/docs/upgrading.md b/docs/upgrading.md index 837c0f9..40541b7 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1511,6 +1511,10 @@ These changes are in 0.35.0. - **A plan made from a `fluid ship --release-dir` release can be applied when the contract has a dbt build.** The release now holds `bundle.tgz.provenance.json`, listed in `release.json` `files` and outside `identity`. Through 0.34.0 `fluid apply` refused such a plan with `apply_plan_digest_bundle_mismatch`. `fluid ship` warns `release_provenance_missing` when it cannot write the file. See [Publishing a release](./cli/ship.md#publishing-a-release). - **A plan can be applied in another checkout of the same tree.** When the paths the plan records are gone, `fluid apply` finds the contract through the provenance file beside the bundle it verified the plan against. The dbt files must still match the packed copy. See [The dbt project](./cli/apply.md#the-dbt-project). +*(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* + +- **`data-product-forge[profile]` installs flux-stats 0.7.0.** The extra asked for `flux-stats>=0.6.2,<0.7` in 0.34.0 and asks for `flux-stats[yaml]>=0.7.0,<0.8` now. Upgrade the extra with `pip install -U 'data-product-forge[profile]'`. `fluid profile` stops with `profile_statistics_unavailable` on an older flux-stats. See [fluid profile](./cli/profile.md). + ## Older releases Upgrading from before `0.15.0`? Start with the From a3d9fc66856291b9bc263f6b9ad59bd539c4bb97 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:04:22 +0100 Subject: [PATCH 4/7] docs: fluid profile, one tense for the old gap and the flux-stats error as 0.35.0 prints it (forge-cli #1021) --- docs/cli/profile.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/cli/profile.md b/docs/cli/profile.md index f71dad0..5be448f 100644 --- a/docs/cli/profile.md +++ b/docs/cli/profile.md @@ -6,7 +6,7 @@ Profile one expose's records where they live, and write a profile document of ou FLUX's statistic registry, `flux-stats`, reads every value and forms every statistic. The fidelity check uses the same registry, so a profile and a check of the same records hold the same numbers. -The `[profile]` extra installs DuckDB and `flux-stats` from PyPI. 0.34.0 needed `>=0.6.2,<0.7`. *(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* It needs `flux-stats[yaml]>=0.7.0,<0.8`. Without flux-stats, `fluid profile` stops before it reads anything and says what to install: +The `[profile]` extra installs DuckDB and `flux-stats` from PyPI: `flux-stats>=0.6.2,<0.7` through 0.34.0 and, *(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))*, `flux-stats[yaml]>=0.7.0,<0.8`. Without flux-stats, `fluid profile` stops before it reads anything and says what to install: ```text ❌ profile_statistics_unavailable [ERR_PROFILE_STATISTICS_UNAVAILABLE] @@ -166,11 +166,11 @@ The block is read the way the FLUX engine reads it, and the fidelity grammar in *(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* `--bundle` reads the bundle's YAML with flux-stats' exact reader and puts its block in the canonical form with flux-stats, where 0.34.0 used a copy of the engine's code. The bounds hold as before and a refusal still names the file, and the line where the reader reports one, never a value. Its words differ in places: a YAML error names the line and column where the reader found it, the most values a document may write out through its aliases is 1,000,000, and the characters it writes out are bounded too. A block whose bins are not strictly increasing grid points is refused before any record is read (`FLX-5004`), as the fidelity check refuses it. -`--bundle` needs flux-stats 0.6.2 or newer, the first release that holds the fidelity grammar, and through 0.34.0 the `[profile]` extra asked for `>=0.6.2,<0.7`. *(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* It asks for `flux-stats[yaml]>=0.7.0,<0.8`, the first release that holds the readers and the check below. With an older flux-stats, `--bundle` stops before it reads anything: +Through 0.34.0, `--bundle` needed flux-stats 0.6.2 or newer, the first release that holds the fidelity grammar, and the `[profile]` extra asked for `>=0.6.2,<0.7`. *(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* Every `fluid profile` run needs `flux-stats[yaml]>=0.7.0,<0.8`, the first release that holds the readers and the check below. With an older flux-stats, `fluid profile` stops before it reads anything, whether or not it has `--bundle`: ```text ❌ profile_statistics_unavailable [ERR_PROFILE_STATISTICS_UNAVAILABLE] - detail: --bundle reads the fidelity block with flux-stats, and the installed flux-stats lacks flux_stats.fidelity. It needs flux-stats[yaml]>=0.7.0,<0.8 + detail: fluid profile forms every statistic with flux-stats, the statistic registry FLUX shares with its fidelity check, and flux-stats 0.6.2 is installed but cannot be imported (ImportError). It needs flux-stats[yaml]>=0.7.0,<0.8; install it with: pip install 'data-product-forge[profile]' read: nothing: no record was read next: pip install 'data-product-forge[profile]' ``` @@ -200,9 +200,9 @@ What the checks leave readable, on purpose: a grouped statistic lists every grou *(since 0.33.0, [forge-cli #967](https://github.com/Agenticstiger/forge-cli/pull/967), [forge-cli #978](https://github.com/Agenticstiger/forge-cli/pull/978))* One thing changes in that paragraph: a histogram or quantile group withheld for its size now brings secondary suppression, as the checks above say, and its siblings are withheld too. A withheld histogram cell still does not. The checks also leave two more things readable, or open: - A published `min` shows that no record lies below it, and a published `max` that none lies above it. That is a range of no records, and no check withholds a `min` or a `max` for it. A quantile group whose value is the smallest or the largest of its records shows the same, and `acro-default` holds zeros disclosive, so the group is withheld for that. -- Through 0.34.0 the checks did not close one gap. A reader can still subtract one histogram's published counts from another's, or from a level histogram's group size, and read a withheld cell, a zero included, to within the rounding. forge-cli plans to close it with FLUX's `check_edge_counts`, which a flux-stats release has to carry first, and to ship that in a later release. +- Through 0.34.0 the checks left one gap: a reader could subtract one histogram's published counts from another's, or from a level histogram's group size, and read a withheld cell, a zero included, to within the rounding. -*(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* Through 0.34.0 the last point above was an open gap. Now `fluid profile` passes every histogram, level histogram and quantile of a column to flux-stats' `check_edge_counts` in one call, and publishes what it lets through. A histogram and a coarser one of one column, or a published cell and the column's count, no longer give a withheld cell back by subtraction, and a cell that does goes with it. A level histogram is checked over its levels as it is over bins, so one small level withholds a neighbour. A histogram that does not sit on the column's one partition of bins is withheld, and so is a group whose rounded counts no longer prove its quantile. The summary counts a cell or group withheld for its size or rounding as `A`, and one withheld so that another cannot be recovered as `D`. The check does not see a column's `support/1`, the group sizes of a statistic grouped by the column, or a declared level that holds no record, which is absent from a grouped statistic. +*(since 0.35.0, [forge-cli #1021](https://github.com/Agenticstiger/forge-cli/pull/1021))* `fluid profile` passes every histogram, level histogram and quantile of a column to flux-stats' `check_edge_counts` in one call, and publishes what it lets through. A histogram and a coarser one of one column, or a published cell and the column's count, no longer give a withheld cell back by subtraction, and a cell that does goes with it. A level histogram is checked over its levels as it is over bins, so one small level withholds a neighbour. A histogram that does not sit on the column's one partition of bins is withheld, and so is a group whose rounded counts no longer prove its quantile. The summary counts a cell or group withheld for its size or rounding as `A`, and one withheld so that another cannot be recovered as `D`. The check does not see a column's `support/1`, the group sizes of a statistic grouped by the column, or a declared level that holds no record, which is absent from a grouped statistic. A `correlation` property is not answered: its moments need the exact number of records, which the checks round. The summary names it, and `fluxsim verify` fails with `FLX-5037`, so a block you check against a profile cannot declare a correlation yet. From 96f278b81856615a0b2bbed55c00fc24df0b669f Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:31:06 +0100 Subject: [PATCH 5/7] docs: forge-cli 0.35.0 (pin, contract reference, release record) --- README.md | 2 +- docs/.vuepress/cli-version.json | 4 ++-- docs/README.md | 2 +- docs/advanced/environment-variables.md | 2 +- docs/advanced/operating-in-ci.md | 4 ++-- docs/cli/README.md | 2 +- docs/cli/contract-tests.md | 2 +- docs/cli/generate.md | 4 ++-- docs/cli/scaffold-ci.md | 2 +- docs/cli/version.md | 2 +- docs/faq/README.md | 2 +- docs/getting-started/README.md | 4 ++-- docs/providers/README.md | 2 +- docs/recipes/per-environment-overlays.md | 4 ++-- docs/reference/contract-0.7.6.md | 2 +- docs/reference/contract-0.7.7-preview.md | 2 +- docs/releases.md | 2 +- docs/sdk-and-plugins/journeys/custom-validator.md | 2 +- docs/sdk-and-plugins/journeys/your-own-scaffolding.md | 2 +- docs/sdk-and-plugins/reference/companion-packages.md | 2 +- docs/sdk-and-plugins/reference/trust-model.md | 2 +- docs/upgrading.md | 8 ++++---- docs/vision.md | 2 +- docs/walkthrough/11-stage-pipeline.md | 4 ++-- docs/walkthrough/jenkins-cicd.md | 4 ++-- docs/walkthrough/local.md | 2 +- scripts/forge-cli-releases.json | 2 +- 27 files changed, 37 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 4e8ddc4..24130ed 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ The site is organised the way a reader uses it (get started, tutorials, how-to g ## Current Versioning -- Current CLI release documented here: `0.34.0` +- Current CLI release documented here: `0.35.0` - Contract schema: `fluidVersion: 0.7.6` is the stable default; `0.7.7` is an opt-in preview Those are different on purpose. `fluid version` reports the installed CLI release, while `fluidVersion` inside a contract selects the contract schema version. diff --git a/docs/.vuepress/cli-version.json b/docs/.vuepress/cli-version.json index 91c4bd9..06c28ce 100644 --- a/docs/.vuepress/cli-version.json +++ b/docs/.vuepress/cli-version.json @@ -1,6 +1,6 @@ { - "supportedCliVersion": "0.34.0", - "supportedCliInstallSpec": "data-product-forge==0.34.0", + "supportedCliVersion": "0.35.0", + "supportedCliInstallSpec": "data-product-forge==0.35.0", "supportedFluidContractVersion": "0.7.6", "quickstartScaffoldVersion": "0.7.5", "supportedSdkVersion": "0.10.0", diff --git a/docs/README.md b/docs/README.md index 09cb9c7..1c59427 100644 --- a/docs/README.md +++ b/docs/README.md @@ -62,7 +62,7 @@ Hello from the my-first-product data product,2026-10-05 10:35:49.479853+02 [Getting Started](./getting-started/README.md) walks through each step with its output, then makes the product read a CSV and adds a data-quality rule. It explains why it uses the `fluid.starter` blueprint rather than `fluid init --quickstart`. -This site documents CLI `0.34.0`, with contract schema `0.7.6` as the stable default and `0.7.7` as an [opt-in preview](./reference/preview-fields.md). `fluid version` is the CLI release you installed; `fluidVersion` inside a contract is the [schema version](./reference/README.md) it is written against. Which version each scaffold writes is in [Understand the version numbers](./getting-started/README.md#understand-the-version-numbers). +This site documents CLI `0.35.0`, with contract schema `0.7.6` as the stable default and `0.7.7` as an [opt-in preview](./reference/preview-fields.md). `fluid version` is the CLI release you installed; `fluidVersion` inside a contract is the [schema version](./reference/README.md) it is written against. Which version each scaffold writes is in [Understand the version numbers](./getting-started/README.md#understand-the-version-numbers). ## Follow one product from laptop to production diff --git a/docs/advanced/environment-variables.md b/docs/advanced/environment-variables.md index 09975ff..4d3e675 100644 --- a/docs/advanced/environment-variables.md +++ b/docs/advanced/environment-variables.md @@ -162,7 +162,7 @@ The CLI itself does not read these; the generated pipeline and the generated DAG | Variable | Read by | Effect | |---|---|---| -| `FLUID_PACKAGE_SPEC` | the generated Jenkins stage 0 | Package spec installed into the workspace venv. Defaults to the generating CLI version with the extras the contract needs, for example `data-product-forge[local]==0.34.0`. | +| `FLUID_PACKAGE_SPEC` | the generated Jenkins stage 0 | Package spec installed into the workspace venv. Defaults to the generating CLI version with the extras the contract needs, for example `data-product-forge[local]==0.35.0`. | | `FLUID_PIP_INDEX_URL`, `FLUID_PIP_EXTRA_INDEX_URL` | the generated install step | Primary and extra pip index. Blank means PyPI. pip takes the highest version across both, so for private packages use one mirror that proxies PyPI in `FLUID_PIP_INDEX_URL` and leave the extra index empty; see [Operating in CI](./operating-in-ci.md#install). | | `FLUID_ALLOW_PRERELEASE` | the generated install step | `true` adds `pip --pre`. | | `FLUID_CONFIG_PATH` | set by generated pipelines | `./fluid_config`. Nothing in the CLI reads it. | diff --git a/docs/advanced/operating-in-ci.md b/docs/advanced/operating-in-ci.md index 2bd4176..192487a 100644 --- a/docs/advanced/operating-in-ci.md +++ b/docs/advanced/operating-in-ci.md @@ -16,7 +16,7 @@ fluid generate ci contract.fluid.yaml --system jenkins --out Jenkinsfile [install-mode: pypi] Jenkinsfile written -> Jenkinsfile |- Jenkins installs FLUID_PACKAGE_SPEC into $WORKSPACE/.fluid-venv (stage 0) ... - |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.34.0 + |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.35.0 |- Jenkins plugins required: workflow-aggregator, git ``` @@ -172,7 +172,7 @@ Jenkins learns a job's parameters from its first build. As measured on a Jenkins ### Install -Stage 0 creates `$WORKSPACE/.fluid-venv`, because agents that follow PEP 668 refuse a bare `pip install`, and installs `FLUID_PACKAGE_SPEC` into it. The default is the version that generated the file with the extras the contract's base and overlays need, for example `data-product-forge[local]==0.34.0`. Every stage then runs `fluid` from that venv, and stage 0 fails if the first `fluid` on `PATH` is another one. The agent needs `python3` with the `venv` module, and `rsync` for a `file://` or `ssh://` stage 11 destination. The workspace is removed with Jenkins's core `deleteDir()`, so the `ws-cleanup` plugin is not needed. +Stage 0 creates `$WORKSPACE/.fluid-venv`, because agents that follow PEP 668 refuse a bare `pip install`, and installs `FLUID_PACKAGE_SPEC` into it. The default is the version that generated the file with the extras the contract's base and overlays need, for example `data-product-forge[local]==0.35.0`. Every stage then runs `fluid` from that venv, and stage 0 fails if the first `fluid` on `PATH` is another one. The agent needs `python3` with the `venv` module, and `rsync` for a `file://` or `ssh://` stage 11 destination. The workspace is removed with Jenkins's core `deleteDir()`, so the `ws-cleanup` plugin is not needed. Pin a different version at build time with `FLUID_PACKAGE_SPEC=data-product-forge==X.Y.Z`. `FLUID_PIP_INDEX_URL` (blank means PyPI), `FLUID_PIP_EXTRA_INDEX_URL` and `FLUID_ALLOW_PRERELEASE` (`true` adds `pip --pre`) cover TestPyPI pilots and private mirrors. With `--install-mode dev-source` the file installs from a `/forge-cli-src` bind mount and fails if the mount is missing; it never falls back to PyPI. diff --git a/docs/cli/README.md b/docs/cli/README.md index 82fb31d..5ce68fd 100644 --- a/docs/cli/README.md +++ b/docs/cli/README.md @@ -1,6 +1,6 @@ # CLI Reference -This section tracks the command surface of `forge-cli` `0.34.0`, the release `pip install data-product-forge` installs. `fluid --help` shows the promoted commands. +This section tracks the command surface of `forge-cli` `0.35.0`, the release `pip install data-product-forge` installs. `fluid --help` shows the promoted commands. ::: tip Looking up *how to do something* (not *what a command does*)? The page you actually want is [**CLI by task →**](./tasks/) — narrative walkthroughs organized around things you might be trying to accomplish, like "deploy to a new cloud", "add quality rules", "debug a failed run", or "add agent governance". Each task page links to the relevant commands. **This** page is the command index, grouped by purpose — better for lookup, worse for onboarding. diff --git a/docs/cli/contract-tests.md b/docs/cli/contract-tests.md index 0d1f6d0..254510a 100644 --- a/docs/cli/contract-tests.md +++ b/docs/cli/contract-tests.md @@ -87,7 +87,7 @@ The contract is loaded the way `fluid plan` loads it, with `$ref` fragments and ```yaml - name: Schema gate run: | - pip install data-product-forge==0.34.0 + pip install data-product-forge==0.35.0 fluid contract-tests contract.fluid.yaml --baseline baseline.schema.json ``` diff --git a/docs/cli/generate.md b/docs/cli/generate.md index 864a595..ddd939f 100644 --- a/docs/cli/generate.md +++ b/docs/cli/generate.md @@ -528,7 +528,7 @@ Options: | `--no-generate-artifacts` | Skip the transformation and schedule stages, for reference-only contracts. Through 0.32.0 it was also set automatically when any build's `pattern` was `hybrid-reference`, `reference` or `external-reference`. It sets stage 3's default to off on every system. *(since 0.33.0, [forge-cli #1000](https://github.com/Agenticstiger/forge-cli/pull/1000))* Nothing sets it automatically. A contract with reference builds, or only reference builds, still runs stage 3. | | `--install-mode {pypi,dev-source}` | 0.30.1: Jenkins and Tekton. `pypi` (default): the install step installs the package spec (Jenkins: `FLUID_PACKAGE_SPEC` into a virtual environment in the workspace). `dev-source` installs from a `/forge-cli-src` bind mount for contributor labs and fails if the mount is missing; on Tekton it also drops the `fluid-package-spec` and pip index parameters from `pipeline.yaml`. *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* Every system reads it. *(since 0.33.0, [forge-cli #1000](https://github.com/Agenticstiger/forge-cli/pull/1000))* `dev-source` runs `pip install -e /forge-cli-src`, which puts `fluid` on `PATH`, instead of exporting `PYTHONPATH`. On the six other systems it stops with a `FATAL` line when `/forge-cli-src/pyproject.toml` is missing, and when the install fails. The runner needs a package index, or `setuptools` and `setuptools-scm` installed with `PIP_NO_BUILD_ISOLATION=0` (pip reads it inverted). Jenkins keeps the agent's installed `fluid` with `PYTHONPATH=/forge-cli-src`. In 0.32.0 every other system exported `PYTHONPATH=/forge-cli-src` in the install step, which ended with the step and provided no `fluid` command, so the install step failed at `fluid --version`. | | `--image IMAGE` | *(since 0.33.0, [forge-cli #1000](https://github.com/Agenticstiger/forge-cli/pull/1000))* Tekton only: the image the Task steps run. Default `python:3.12-slim`, pinned by digest. It takes `[registry[:port]/]name[:tag][@sha256:]`, and a value that is not an image reference is refused with `generate_ci_invalid_option`. Pin yours by digest too (`name:tag@sha256:`): a tag can move. Another system refuses the flag with `generate_ci_image_unsupported`. | -| `--fluid-package-spec SPEC` | Default of the `FLUID_PACKAGE_SPEC` parameter. Default: the version that generated the file, with the extras the contract's bindings need across its base and every overlay, such as `data-product-forge[local]==0.34.0` or `data-product-forge[aws,local]==0.34.0`. | +| `--fluid-package-spec SPEC` | Default of the `FLUID_PACKAGE_SPEC` parameter. Default: the version that generated the file, with the extras the contract's bindings need across its base and every overlay, such as `data-product-forge[local]==0.35.0` or `data-product-forge[aws,local]==0.35.0`. | | `--fluid-env-default ENV` | Default of `FLUID_ENV`, the overlay the stages pass to `--env`, and the fallback the stages read, so a build with no parameters runs in `ENV`. Letters, digits, `.`, `_` and `-`, starting with a letter. Default `dev`. | | `--apply-mode-default MODE` | Default of `APPLY_MODE`, the mode stage 6 plans and stage 7 applies: `dry-run`, `create-only`, `amend`, `amend-and-build`, `replace` or `replace-and-build`. Default `dry-run` for Jenkins, `amend` for Tekton. | | `--default-publish-target TARGET` | Default of `PUBLISH_TARGETS`, and the fallback stage 10 reads when a build has no parameters. Default `datamesh-manager`. Pick the catalog your team publishes to, for example `fluid-command-center`, `horizon`, `datahub` or `collibra`. | @@ -566,7 +566,7 @@ Below is the 0.30.1 measurement. As measured on 0.18.1 by generating each system *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* Every stage runs `fluid ship "$CONTRACT" --stage N`, and the header of the file says: "A change request (pull request build) stops after stage 6. A branch build applies after a person approves (APPLY_APPROVAL, an input step), one build per environment at a time (lockable-resources)." Stages 7 to 11 sit behind an `Approve apply` stage with an `input` step and a `milestone`, and then run under a `lock` named `fluid-apply-`. They run on the default branch only (`main`, or `--default-branch`), or in a job that builds no branch. `APPLY_APPROVAL` defaults to `true`: set it to `false` for an unattended job. A change request build has no applied-plan baseline for stage 5, because its own job never applies. The sections below describe the 0.30.1 file. -- **Install.** Stage 0 creates `$WORKSPACE/.fluid-venv` and installs `FLUID_PACKAGE_SPEC` into it, because an agent whose system Python is externally managed (PEP 668) refuses a bare `pip install`. The spec defaults to the version that generated the file, so a Jenkinsfile generated by 0.34.0 installs `data-product-forge[local]==0.34.0` and does not move when a new release ships. Override it per build with the `FLUID_PACKAGE_SPEC` parameter, `FLUID_PIP_INDEX_URL` and `FLUID_PIP_EXTRA_INDEX_URL` (TestPyPI), or `FLUID_ALLOW_PRERELEASE`. pip takes the highest version across every index it is given, so do not pair a test or private index with PyPI; see [Operating in CI](../advanced/operating-in-ci.md#install). +- **Install.** Stage 0 creates `$WORKSPACE/.fluid-venv` and installs `FLUID_PACKAGE_SPEC` into it, because an agent whose system Python is externally managed (PEP 668) refuses a bare `pip install`. The spec defaults to the version that generated the file, so a Jenkinsfile generated by 0.35.0 installs `data-product-forge[local]==0.35.0` and does not move when a new release ships. Override it per build with the `FLUID_PACKAGE_SPEC` parameter, `FLUID_PIP_INDEX_URL` and `FLUID_PIP_EXTRA_INDEX_URL` (TestPyPI), or `FLUID_ALLOW_PRERELEASE`. pip takes the highest version across every index it is given, so do not pair a test or private index with PyPI; see [Operating in CI](../advanced/operating-in-ci.md#install). - **Plugins.** `workflow-aggregator` and `git`, plus `copyartifact` with `--diff-last-applied`. *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* Also `lockable-resources` and `junit`. The workspace is removed with the core `deleteDir()` step, so the `ws-cleanup` plugin is not needed. - **The first build.** Jenkins learns a Pipeline's parameters from its first build, so a `buildWithParameters` request to a job that has never run is refused (observed on a Jenkins controller on 4 Oct 2026). Start the job once from the UI with no parameters. The generated file is written for that run: every parameter default is also the fallback each stage reads, so a first build, or the first build after a restart that re-seeds jobs from job-dsl or JCasC, runs what the declarations say. `--fluid-env-default` and `--apply-mode-default` set what that run does. - **Stage 10.** `fluid publish` receives `--env "${FLUID_ENV:-dev}"` by default, so the catalog gets the contract with the overlay stages 5 to 9 used. Before 0.16.3 `fluid publish` had no `--env`, and the default was off. Pass `--no-publish-include-env` only when the pipeline installs a CLI older than `fluid publish --env`. *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* The flag is removed, and ship always passes `--env`. GitLab CI, GitHub Actions, Azure DevOps and Bitbucket pass `--env "${FLUID_ENV:-dev}"` to `fluid publish` in their default output; the default CircleCI output has no `fluid publish` step. diff --git a/docs/cli/scaffold-ci.md b/docs/cli/scaffold-ci.md index fd8188b..e035359 100644 --- a/docs/cli/scaffold-ci.md +++ b/docs/cli/scaffold-ci.md @@ -45,7 +45,7 @@ The generated file installs `data-product-forge` from PyPI (`pip install --quiet | publish | On `main` with `DMM_API_URL` set: `fluid publish $CONTRACT --catalog $CATALOG`. | ::: warning Pin the version in a pipeline that holds credentials -An unpinned `pip install data-product-forge` takes whatever release is latest when the job runs, and the same job holds the credentials the apply stage uses. After generating, edit the install line to pin the version you tested, for example `pip install --quiet "data-product-forge==0.34.0"`, and move the pin deliberately when you upgrade. [`fluid generate ci`](./generate.md#fluid-generate-ci) pins the install to the version that generated the file. +An unpinned `pip install data-product-forge` takes whatever release is latest when the job runs, and the same job holds the credentials the apply stage uses. After generating, edit the install line to pin the version you tested, for example `pip install --quiet "data-product-forge==0.35.0"`, and move the pin deliberately when you upgrade. [`fluid generate ci`](./generate.md#fluid-generate-ci) pins the install to the version that generated the file. ::: The settings are `CONTRACT` (the path you passed), `PROVIDER` (default `default`), `BUILD_ID` (default empty; set it to a `builds[].id` to run that build), `CATALOG` (default `datamesh-manager`), `AIRFLOW_DAGS_DEST` and `DMM_API_URL`. GitLab reads them as CI/CD variables, GitHub Actions as `env`, repository `vars` and `secrets`, Jenkins from the `environment` block. diff --git a/docs/cli/version.md b/docs/cli/version.md index 1311381..2eefc12 100644 --- a/docs/cli/version.md +++ b/docs/cli/version.md @@ -69,7 +69,7 @@ Python: 3.11.15 (darwin) System: Darwin arm64 ``` -`fluid --version` is the top-level flag and prints one line, `FLUID Forge CLI v0.34.0`. +`fluid --version` is the top-level flag and prints one line, `FLUID Forge CLI v0.35.0`. ## Options diff --git a/docs/faq/README.md b/docs/faq/README.md index 1c3adf1..8152299 100644 --- a/docs/faq/README.md +++ b/docs/faq/README.md @@ -58,7 +58,7 @@ pip install --upgrade "data-product-forge[local]" fluid version ``` -Then follow the [upgrade guide](../upgrading.md). It has a checklist for each version you cross, because some upgrades need action: `0.17.0`, for example, moves remote state to a per-provider key for a contract that uses a per-contract state key, and needs one `fluid schedule-sync` to retire old Airflow DAGs if you sync Airflow DAGs, and `0.16.3` changed some results on purpose. Pin an exact version in CI (`data-product-forge==0.34.0`), not a range. +Then follow the [upgrade guide](../upgrading.md). It has a checklist for each version you cross, because some upgrades need action: `0.17.0`, for example, moves remote state to a per-provider key for a contract that uses a per-contract state key, and needs one `fluid schedule-sync` to retire old Airflow DAGs if you sync Airflow DAGs, and `0.16.3` changed some results on purpose. Pin an exact version in CI (`data-product-forge==0.35.0`), not a range. Your contracts do not need a new `fluidVersion`: `0.27.0` reads `0.7.1` to `0.7.6`, and `0.7.7` as a preview. diff --git a/docs/getting-started/README.md b/docs/getting-started/README.md index 7c21f27..b4699fa 100644 --- a/docs/getting-started/README.md +++ b/docs/getting-started/README.md @@ -25,7 +25,7 @@ fluid version ```text ╭───────────────────────────────────── 📦 Version Information ─────────────────────────────────────╮ │ FLUID CLI │ -│ Version: 0.34.0 │ +│ Version: 0.35.0 │ │ API: v1 │ │ │ │ Supported Specifications: │ @@ -55,7 +55,7 @@ The table lists the extras a first project needs. A base install has none of `li ```bash pip install "data-product-forge[local,gcp]" # several extras at once -pip install "data-product-forge[local]==0.34.0" # an exact version, for CI +pip install "data-product-forge[local]==0.35.0" # an exact version, for CI ``` A cloud `fluid apply` runs OpenTofu, so it also needs `tofu` 1.6.0 or newer on your `PATH`, or `fluid apply --ensure-opentofu`, which downloads a pinned build from the OpenTofu release and checks it against that release's `SHA256SUMS` (integrity, not a signature). *(since 0.31.0, [forge-cli #899](https://github.com/Agenticstiger/forge-cli/pull/899))* The minimum is 1.12.0, and the pinned build is 1.13.9. For production CI, install OpenTofu with a cosign- or gpg-verified install instead, and keep `tofu` on the runner image. The local path in this tutorial does not use OpenTofu. diff --git a/docs/providers/README.md b/docs/providers/README.md index 81879d2..fa2eb11 100644 --- a/docs/providers/README.md +++ b/docs/providers/README.md @@ -8,7 +8,7 @@ Fluid Forge uses one contract format across local and provider-backed execution ## Docs baseline -- CLI release covered by the primary docs: `0.34.0` +- CLI release covered by the primary docs: `0.35.0` - Contract schema: `0.7.6` is the current stable version, and the default since `0.26.0`; `0.7.7` is a preview you opt into with `fluidVersion: "0.7.7"`. The GCP retention, encryption and `binding.principals` fields need `0.7.6`. - Which `fluidVersion` each scaffolding command writes is listed in [`fluid init`](../cli/init.md#which-fluidversion-each-path-writes). diff --git a/docs/recipes/per-environment-overlays.md b/docs/recipes/per-environment-overlays.md index 2116e3c..b374365 100644 --- a/docs/recipes/per-environment-overlays.md +++ b/docs/recipes/per-environment-overlays.md @@ -249,14 +249,14 @@ jobs: matrix: env: [dev, staging, prod] steps: - - run: pip install "data-product-forge==0.34.0" + - run: pip install "data-product-forge==0.35.0" - run: fluid validate contract.fluid.yaml --env ${{ matrix.env }} - run: fluid plan contract.fluid.yaml --env ${{ matrix.env }} ``` Pin the version: an unpinned `pip install` in a pipeline takes whatever is newest on the day. Add the extras your clouds need, for example -`"data-product-forge[gcp]==0.34.0"`. +`"data-product-forge[gcp]==0.35.0"`. ## See also diff --git a/docs/reference/contract-0.7.6.md b/docs/reference/contract-0.7.6.md index e622585..eabe564 100644 --- a/docs/reference/contract-0.7.6.md +++ b/docs/reference/contract-0.7.6.md @@ -9,7 +9,7 @@ description: "The fields of the FLUID 0.7.6 contract schema (stable): path, type Status: **stable**. Schema 0.7.6 is the newest stable schema bundled with the CLI. The newer 0.7.7 schema is a preview; see the [0.7.7 preview delta](./contract-0.7.7-preview.md). -Generated from `fluid_build/schemas/fluid-schema-0.7.6.json` in the `data-product-forge` 0.34.0 package (sha256 `61ca91a63087b89b`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`. +Generated from `fluid_build/schemas/fluid-schema-0.7.6.json` in the `data-product-forge` 0.35.0 package (sha256 `61ca91a63087b89b`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`. Each table row is one field, addressed by its path from the contract root: `.` descends into an object, `[]` marks the items of a list, and `` stands for any key of a map. **Required** means required inside its parent object; the parent can itself be optional. **Allowed values** lists the enum, pattern and range the schema enforces. An object typed plainly `object` rejects keys the schema does not list; `object (open)` accepts extra keys and `object (free-form)` has no listed keys. A row that begins "Only when ..." exists only under that condition. See [how to read this reference](./README.md#how-to-read-a-row). diff --git a/docs/reference/contract-0.7.7-preview.md b/docs/reference/contract-0.7.7-preview.md index f701943..8b559a7 100644 --- a/docs/reference/contract-0.7.7-preview.md +++ b/docs/reference/contract-0.7.7-preview.md @@ -9,7 +9,7 @@ description: "What the FLUID 0.7.7 preview schema adds to, and changes in, the s Status: **preview**. A contract opts in with `fluidVersion: "0.7.7"`; a contract that declares 0.7.6 is unaffected. This page lists the difference between the 0.7.6 schema and the 0.7.7 schema as bundled with the CLI: added fields, fields whose definition changed, and fields removed. A field not listed here has the same type, required flag, allowed values, default and description in both; for those, use the [0.7.6 reference](./contract-0.7.6.md). A definition used at several paths is listed once, at its first path. -Generated from `fluid_build/schemas/fluid-schema-0.7.7.json` in the `data-product-forge` 0.34.0 package (sha256 `34e0ff10e1b59f46`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`. +Generated from `fluid_build/schemas/fluid-schema-0.7.7.json` in the `data-product-forge` 0.35.0 package (sha256 `34e0ff10e1b59f46`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`. Each table row is one field, addressed by its path from the contract root: `.` descends into an object, `[]` marks the items of a list, and `` stands for any key of a map. **Required** means required inside its parent object; the parent can itself be optional. **Allowed values** lists the enum, pattern and range the schema enforces. An object typed plainly `object` rejects keys the schema does not list; `object (open)` accepts extra keys and `object (free-form)` has no listed keys. A row that begins "Only when ..." exists only under that condition. See [how to read this reference](./README.md#how-to-read-a-row). diff --git a/docs/releases.md b/docs/releases.md index bafa527..da1d12f 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -5,7 +5,7 @@ description: Every forge-cli release, newest first, with what changed and where # Releases -These docs are verified against CLI `0.34.0`. Moving to a newer CLI? See [Upgrading](./upgrading.md). +These docs are verified against CLI `0.35.0`. Moving to a newer CLI? See [Upgrading](./upgrading.md). Each version links to its upgrade steps, its release notes, or the forge-cli CHANGELOG. For what `fluid version`, `fluidVersion` and the schema versions mean, see [Understand the version numbers](./getting-started/README.md#understand-the-version-numbers). diff --git a/docs/sdk-and-plugins/journeys/custom-validator.md b/docs/sdk-and-plugins/journeys/custom-validator.md index 9302692..2eb60f2 100644 --- a/docs/sdk-and-plugins/journeys/custom-validator.md +++ b/docs/sdk-and-plugins/journeys/custom-validator.md @@ -459,7 +459,7 @@ Three places this typically gets installed: ``` 3. **CI** — the bundle from [your-own-ci](./your-own-ci.md) already has a `validate` stage. Add `my-org-validators` to the `pip install` line: ```yaml - - run: pip install --index-url "" "data-product-forge==0.34.0" "my-org-validators==" + - run: pip install --index-url "" "data-product-forge==0.35.0" "my-org-validators==" - run: fluid validate contract.fluid.yaml --strict env: FLUID_PLUGINS_ALLOWLIST: "steward-required,cost-center-required,classification-from-vocab,local" diff --git a/docs/sdk-and-plugins/journeys/your-own-scaffolding.md b/docs/sdk-and-plugins/journeys/your-own-scaffolding.md index 63f402c..0b4dcc2 100644 --- a/docs/sdk-and-plugins/journeys/your-own-scaffolding.md +++ b/docs/sdk-and-plugins/journeys/your-own-scaffolding.md @@ -176,7 +176,7 @@ keywords = [ dependencies = [ "pydantic>=2.0", - "data-product-forge==0.34.0", + "data-product-forge==0.35.0", ] [project.optional-dependencies] diff --git a/docs/sdk-and-plugins/reference/companion-packages.md b/docs/sdk-and-plugins/reference/companion-packages.md index a85f924..a56c8b8 100644 --- a/docs/sdk-and-plugins/reference/companion-packages.md +++ b/docs/sdk-and-plugins/reference/companion-packages.md @@ -48,7 +48,7 @@ The PyPI name reflects the product brand (`data-product-forge`). The import path ```toml dependencies = [ - "data-product-forge==0.34.0", # pin exact for reproducibility + "data-product-forge==0.35.0", # pin exact for reproducibility ] ``` diff --git a/docs/sdk-and-plugins/reference/trust-model.md b/docs/sdk-and-plugins/reference/trust-model.md index 9ac5bf7..e50a8ee 100644 --- a/docs/sdk-and-plugins/reference/trust-model.md +++ b/docs/sdk-and-plugins/reference/trust-model.md @@ -177,7 +177,7 @@ Plugins are tested, scanned, and approved by your platform team before showing u ```toml # In your deploy environment's pyproject.toml or requirements: -data-product-forge==0.34.0 +data-product-forge==0.35.0 data-product-forge-sdk==0.10.0 data-product-forge-custom-scaffold==0.4.1 my-org-validators==1.2.0 diff --git a/docs/upgrading.md b/docs/upgrading.md index 40541b7..9f95c3b 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -1,6 +1,6 @@ --- title: Upgrading -description: How to upgrade and pin data-product-forge, the checklist for each step from 0.15.x to 0.34.0, and what moving the local engine from DuckDB 1.x to 2.x changes, including what breaks, what to do, and how to check it. +description: How to upgrade and pin data-product-forge, the checklist for each step from 0.15.x to 0.35.0, and what moving the local engine from DuckDB 1.x to 2.x changes, including what breaks, what to do, and how to check it. --- # Upgrading @@ -19,7 +19,7 @@ fluid version $ fluid version ╭─────────────────────────── 📦 Version Information ───────────────────────────╮ │ FLUID CLI │ -│ Version: 0.34.0 │ +│ Version: 0.35.0 │ │ API: v1 │ │ │ │ Supported Specifications: │ @@ -40,7 +40,7 @@ DuckDB. **Pin an exact version in CI and production:** ```bash -pip install "data-product-forge[local]==0.34.0" +pip install "data-product-forge[local]==0.35.0" ``` A range such as `>=0.16,<0.17` is not safe for CLI behaviour. Patch releases have changed @@ -56,7 +56,7 @@ through the checklist below. ```console $ fluid generate ci contract.fluid.yaml --system jenkins ... - |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.34.0 + |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.35.0 ``` Upgrading the CLI on your machine does not upgrade that pipeline. Regenerate it, or set diff --git a/docs/vision.md b/docs/vision.md index 539286b..5255d85 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -149,7 +149,7 @@ fluid market --search "customer analytics" ## Versioning in the docs -- Current CLI release baseline: `0.34.0` +- Current CLI release baseline: `0.35.0` - Current scaffolded contract examples: `fluidVersion: 0.7.5` That split is intentional. The CLI release and the contract schema version move on related but different timelines. diff --git a/docs/walkthrough/11-stage-pipeline.md b/docs/walkthrough/11-stage-pipeline.md index e68f80b..1f2fdc1 100644 --- a/docs/walkthrough/11-stage-pipeline.md +++ b/docs/walkthrough/11-stage-pipeline.md @@ -20,7 +20,7 @@ fluid generate ci contract.fluid.yaml --system jenkins --out Jenkinsfile | FLUID_PIP_INDEX_URL = 'https://test.pypi.org/simple/' (TestPyPI) | FLUID_PIP_EXTRA_INDEX_URL = 'https://pypi.org/simple/' (fallback) | FLUID_ALLOW_PRERELEASE = true (pip --pre, alpha/rc releases) - |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.34.0 + |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.35.0 |- Jenkins plugins required: workflow-aggregator, git ``` @@ -64,7 +64,7 @@ Jenkins registers a Pipeline job's `parameters` block when the job first runs, s - `APPLY_MODE` is `dry-run`. Stage 6 plans for `dry-run` and stage 7 renders the plan without changing anything. - Stage 8 runs `fluid policy-apply --mode check`, and stage 9 prints that stage 7 ran as a dry run and skips. - Stage 10 (`RUN_STAGE_10_PUBLISH`) and stage 11 (`RUN_STAGE_11_SCHEDULE_SYNC`) are off. Turned on, they print the same dry-run message and skip. -- `CONTRACT` is the path you generated from, `FLUID_ENV` is `dev`, and `FLUID_PACKAGE_SPEC` is the CLI version that generated the file with the extras the contract needs: `data-product-forge[local]==0.34.0` for this contract. +- `CONTRACT` is the path you generated from, `FLUID_ENV` is `dev`, and `FLUID_PACKAGE_SPEC` is the CLI version that generated the file with the extras the contract needs: `data-product-forge[local]==0.35.0` for this contract. Run one plain build first, then use Build with Parameters. Jenkins refuses `buildWithParameters` for a job that has never run, because it has not learned the job's parameters yet (observed on a Jenkins controller on 4 to 5 October 2026 with a pipeline generated by this command). diff --git a/docs/walkthrough/jenkins-cicd.md b/docs/walkthrough/jenkins-cicd.md index 0b269fb..3f94f07 100644 --- a/docs/walkthrough/jenkins-cicd.md +++ b/docs/walkthrough/jenkins-cicd.md @@ -20,7 +20,7 @@ fluid generate ci contract.fluid.yaml --system jenkins --out Jenkinsfile | FLUID_PIP_INDEX_URL = 'https://test.pypi.org/simple/' (TestPyPI) | FLUID_PIP_EXTRA_INDEX_URL = 'https://pypi.org/simple/' (fallback) | FLUID_ALLOW_PRERELEASE = true (pip --pre, alpha/rc releases) - |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.34.0 + |- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.35.0 |- Jenkins plugins required: workflow-aggregator, git ``` @@ -28,7 +28,7 @@ fluid generate ci contract.fluid.yaml --system jenkins --out Jenkinsfile pip picks the highest version of a package across every index it is given. With `FLUID_PIP_INDEX_URL` set to TestPyPI and `FLUID_PIP_EXTRA_INDEX_URL` set to PyPI, as the output above suggests, whichever index holds the higher version wins, for `data-product-forge` and for each of its dependencies. Anyone can register a name on TestPyPI, so a pilot build can install a package you did not choose. Run TestPyPI pilots only on an agent that holds no deploy credentials. For private packages, set `FLUID_PIP_INDEX_URL` to one mirror that proxies PyPI and leave `FLUID_PIP_EXTRA_INDEX_URL` empty. ::: -Commit the `Jenkinsfile` beside the contract. The file is generated once and then belongs to you: after you upgrade the CLI, regenerate it and review the diff to pick up changes. The `CONTRACT` parameter defaults to the path you generated from, and `FLUID_PACKAGE_SPEC` to the CLI version that generated the file with the extras the contract and its overlays need. A contract that has an `overlays/gcp.yaml` binding to BigQuery generates `data-product-forge[gcp,local]==0.34.0`; `--fluid-package-spec` overrides it. +Commit the `Jenkinsfile` beside the contract. The file is generated once and then belongs to you: after you upgrade the CLI, regenerate it and review the diff to pick up changes. The `CONTRACT` parameter defaults to the path you generated from, and `FLUID_PACKAGE_SPEC` to the CLI version that generated the file with the extras the contract and its overlays need. A contract that has an `overlays/gcp.yaml` binding to BigQuery generates `data-product-forge[gcp,local]==0.35.0`; `--fluid-package-spec` overrides it. ## Set up Jenkins diff --git a/docs/walkthrough/local.md b/docs/walkthrough/local.md index 82687c3..3470d21 100644 --- a/docs/walkthrough/local.md +++ b/docs/walkthrough/local.md @@ -44,7 +44,7 @@ The `local` extra installs DuckDB (`duckdb>=1.5.0`) and pandas. The bare package ```text ╭───────────────────────────────────── 📦 Version Information ─────────────────────────────────────╮ │ FLUID CLI │ -│ Version: 0.34.0 │ +│ Version: 0.35.0 │ │ API: v1 │ │ │ │ Supported Specifications: │ diff --git a/scripts/forge-cli-releases.json b/scripts/forge-cli-releases.json index 744f01b..a295a3f 100644 --- a/scripts/forge-cli-releases.json +++ b/scripts/forge-cli-releases.json @@ -21,7 +21,7 @@ ], "repo": "Agenticstiger/forge-cli", "changelogUrl": "https://raw.githubusercontent.com/Agenticstiger/forge-cli/main/CHANGELOG.md", - "reviewedThroughRelease": "0.34.0", + "reviewedThroughRelease": "0.35.0", "releaseOfPr": { "707": "0.20.0", "709": "0.20.0", From fd406988b1db51627e528a2a3f00dd115e5d6a1c Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:31:43 +0100 Subject: [PATCH 6/7] docs: fluid ship opens with the 11-stage runner, the four-stage macro moves to history --- docs/cli/ship.md | 180 ++++++++++++++++++++++++----------------------- 1 file changed, 92 insertions(+), 88 deletions(-) diff --git a/docs/cli/ship.md b/docs/cli/ship.md index 516d7c3..99e6c28 100644 --- a/docs/cli/ship.md +++ b/docs/cli/ship.md @@ -1,8 +1,8 @@ # `fluid ship` -Happy-path macro that chains `validate → bundle → plan → apply` in one command. Use it when you've already iterated on a contract locally and just want to push it through. +Run a contract through the 11-stage pipeline in one command: bundle, validate, generate artifacts, validate artifacts, diff, plan, apply, policy-apply and verify, with publish and schedule-sync when you ask for them. It runs the same way on a laptop and in CI. -*(since 0.31.0, [forge-cli #917](https://github.com/Agenticstiger/forge-cli/pull/917))* `fluid ship` runs the 11-stage pipeline, the same way on a laptop and in CI, in place of the four-stage macro this page describes for the pinned release. See [The 11-stage runner in 0.31.0](#the-11-stage-runner-in-0-31-0). +*(since 0.31.0, [forge-cli #917](https://github.com/Agenticstiger/forge-cli/pull/917))* `fluid ship` runs the 11-stage pipeline. Through 0.30.1 it was a four-stage macro that chained `validate → bundle → plan → apply`; that page text is kept in [The four-stage macro (0.30.1 and earlier)](#the-four-stage-macro-0-30-1-and-earlier). See [The 11-stage runner in 0.31.0](#the-11-stage-runner-in-0-31-0). ::: tip Where this fits `fluid ship` landed in `0.8.3` alongside the day-2 ops surface. @@ -14,92 +14,6 @@ Happy-path macro that chains `validate → bundle → plan → apply` in one com fluid ship [contract-path] [options] ``` -## Examples - -These are the four-stage macro's examples (0.30.1 and earlier). The first exits 2 in 0.31.0, where `--skip-plan` is removed; see [The 11-stage runner in 0.31.0](#the-11-stage-runner-in-0-31-0). - -```bash -# Local dev: validate + apply only (0.30.1 and earlier) -fluid ship --skip-bundle --skip-plan --yes - -# Production-style: validate strictly, then deploy the prod overlay -fluid ship contract.fluid.yaml --strict --env prod --yes - -# Render the apply without writing anything -fluid ship contract.fluid.yaml --env dev --dry-run -``` - -The last command runs all four stages. Each stage line shows the command `ship` ran: - -```text -🚢 fluid ship — running 4 stage(s) -... -[ship validate] $ fluid validate /.../contract.fluid.yaml -[ship bundle] $ fluid bundle /.../contract.fluid.yaml --format tgz -[ship plan] $ fluid plan /.../contract.fluid.yaml --env dev -[ship apply] $ fluid apply /.../contract.fluid.yaml --env dev --dry-run -... -🎉 Ship complete — all 4 stage(s) passed -``` - -## Behavior - -`fluid ship` runs the four core stages in sequence, stopping at the first failure and relaying that stage's exit code: - -```text -1. fluid validate [--strict if --strict is set] -2. fluid bundle --format tgz (skipped if --skip-bundle) -3. fluid plan [--env] (skipped if --skip-plan) -4. fluid apply [--env] [--yes] [--dry-run] -``` - -Each stage runs as a separate process (`python -m fluid_build.cli `), so a stage's own flag handling and errors are unchanged. - -Three things `ship` does not do, which differ from the [11-stage pipeline](../walkthrough/11-stage-pipeline.md): - -- **`--env` reaches only `plan` and `apply`.** `validate` and `bundle` run against the base contract, so the bundle is not built for the environment you apply. -- **`apply` gets the contract path, not the bundle or `plan.json`.** `ship` therefore has no bundle or plan digest binding between the reviewed plan and the apply. For a chain that binds them, use [`fluid generate ci`](./generate.md#fluid-generate-ci) or run the stages yourself. -- **`--dry-run` does not stop before `apply`.** It passes `--dry-run` to `fluid apply`, which renders what it would do and writes nothing. The run still reports every stage as passed. - -## Options - -| Option | Description | -|---|---| -| `[contract-path]` | Path to the contract. Auto-discovered from the current directory if omitted. | -| `--env ` | Environment overlay, passed to `plan` and `apply` only. | -| `--strict` | Pass `--strict` to `fluid validate` (treat warnings as errors). | -| `--yes`, `-y` | Pass `--yes` to `fluid apply` (skip the interactive confirmation). Needed in CI. | -| `--skip-bundle` | Skip the bundle stage. | -| `--skip-plan` | Skip the plan stage. | -| `--dry-run` | Pass `--dry-run` to `fluid apply`. | - -## Files `ship` writes - -The bundle stage runs `fluid bundle --format tgz` with no `--out`, so the bundle goes next to the contract as `.fluid.bundle.tgz`. The plan stage writes `plan.json` in the current directory. Neither is read by `apply`. Add them to `.gitignore`, or pass `--skip-bundle` and `--skip-plan`. - -A `fragments/` directory next to the contract does not change where the bundle goes since 0.26.0 ([forge-cli #763](https://github.com/Agenticstiger/forge-cli/pull/763)). In 0.25.0 and earlier, `fluid bundle` sent a no-`--out` run on a fragment layout to `/contract.bundled.fluid.yaml`, and for `--format tgz` wrote the gzip tarball under that `.yaml` name, which `fluid validate` and `fluid plan` could not read as a contract. If an earlier `fluid ship` left that file beside your contract, delete it. - -## When to use the macro vs. the individual stages - -| Use `fluid ship` when | Use individual stages when | -|---|---| -| You've already iterated and want a clean push | You're debugging a failing stage | -| For demo and quickstart speed | You need stage-specific flags `ship` doesn't expose (per-stage `--out` paths, `bundle --format`, `plan --html`) | -| | You want to inspect `plan.json` before apply, or bind the apply to the plan | - -In 0.30.1 and earlier, for the production-grade 11-stage pipeline (cryptographic plan binding, drift gating, supply-chain signing), use [`fluid generate ci`](./generate.md#fluid-generate-ci) to emit the full pipeline for your CI system instead of `fluid ship`. *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* The pipeline `fluid generate ci` writes calls `fluid ship` for every stage, so the macro and the generated pipeline run the same code: see [What a generated pipeline runs](../advanced/operating-in-ci.md#what-a-generated-pipeline-runs). - -## Exit codes - -`fluid ship` relays the exit code of the **first failing stage**: - -| Code | Meaning | -|---|---| -| `0` | All stages passed | -| Non-zero | Exit code of the failing stage (e.g. `1` from `validate` for a schema error) | - -With no contract path and no `contract.fluid.yaml` in the current directory, `ship` exits `1` with `contract_required`. Stage logs are written to stdout and stderr unmodified: the macro doesn't buffer or reformat them. Pipe to `tee` or to your CI system's log capture as usual. - ## The 11-stage runner in 0.31.0 *(since 0.31.0, [forge-cli #917](https://github.com/Agenticstiger/forge-cli/pull/917))* @@ -391,6 +305,96 @@ A contract outside the current directory (`fluid ship /tmp/x/contract.fluid.yaml - The bundle is `runtime/bundle.tgz` and the plan `runtime/plan.json`, not `.fluid.bundle.tgz` beside the contract and `plan.json` in the current directory. - The exit status is ship's own. A script that read the failing stage's code from `$?` reads it from the report. +## The four-stage macro (0.30.1 and earlier) + +Through 0.30.1, `fluid ship` chained `validate → bundle → plan → apply` and nothing else. The text below describes that macro. In 0.31.0 and later, `--skip-plan` exits 2 and `--skip-bundle` is ignored; see [From the four-stage macro](#from-the-four-stage-macro). + +### Examples of the four-stage macro + +These are the four-stage macro's examples (0.30.1 and earlier). The first exits 2 in 0.31.0, where `--skip-plan` is removed; see [The 11-stage runner in 0.31.0](#the-11-stage-runner-in-0-31-0). + +```bash +# Local dev: validate + apply only (0.30.1 and earlier) +fluid ship --skip-bundle --skip-plan --yes + +# Production-style: validate strictly, then deploy the prod overlay +fluid ship contract.fluid.yaml --strict --env prod --yes + +# Render the apply without writing anything +fluid ship contract.fluid.yaml --env dev --dry-run +``` + +The last command runs all four stages. Each stage line shows the command `ship` ran: + +```text +🚢 fluid ship — running 4 stage(s) +... +[ship validate] $ fluid validate /.../contract.fluid.yaml +[ship bundle] $ fluid bundle /.../contract.fluid.yaml --format tgz +[ship plan] $ fluid plan /.../contract.fluid.yaml --env dev +[ship apply] $ fluid apply /.../contract.fluid.yaml --env dev --dry-run +... +🎉 Ship complete — all 4 stage(s) passed +``` + +### How the four-stage macro behaves + +`fluid ship` runs the four core stages in sequence, stopping at the first failure and relaying that stage's exit code: + +```text +1. fluid validate [--strict if --strict is set] +2. fluid bundle --format tgz (skipped if --skip-bundle) +3. fluid plan [--env] (skipped if --skip-plan) +4. fluid apply [--env] [--yes] [--dry-run] +``` + +Each stage runs as a separate process (`python -m fluid_build.cli `), so a stage's own flag handling and errors are unchanged. + +Three things `ship` does not do, which differ from the [11-stage pipeline](../walkthrough/11-stage-pipeline.md): + +- **`--env` reaches only `plan` and `apply`.** `validate` and `bundle` run against the base contract, so the bundle is not built for the environment you apply. +- **`apply` gets the contract path, not the bundle or `plan.json`.** `ship` therefore has no bundle or plan digest binding between the reviewed plan and the apply. For a chain that binds them, use [`fluid generate ci`](./generate.md#fluid-generate-ci) or run the stages yourself. +- **`--dry-run` does not stop before `apply`.** It passes `--dry-run` to `fluid apply`, which renders what it would do and writes nothing. The run still reports every stage as passed. + +### Options of the four-stage macro + +| Option | Description | +|---|---| +| `[contract-path]` | Path to the contract. Auto-discovered from the current directory if omitted. | +| `--env ` | Environment overlay, passed to `plan` and `apply` only. | +| `--strict` | Pass `--strict` to `fluid validate` (treat warnings as errors). | +| `--yes`, `-y` | Pass `--yes` to `fluid apply` (skip the interactive confirmation). Needed in CI. | +| `--skip-bundle` | Skip the bundle stage. | +| `--skip-plan` | Skip the plan stage. | +| `--dry-run` | Pass `--dry-run` to `fluid apply`. | + +### Files the four-stage macro writes + +The bundle stage runs `fluid bundle --format tgz` with no `--out`, so the bundle goes next to the contract as `.fluid.bundle.tgz`. The plan stage writes `plan.json` in the current directory. Neither is read by `apply`. Add them to `.gitignore`, or pass `--skip-bundle` and `--skip-plan`. + +A `fragments/` directory next to the contract does not change where the bundle goes since 0.26.0 ([forge-cli #763](https://github.com/Agenticstiger/forge-cli/pull/763)). In 0.25.0 and earlier, `fluid bundle` sent a no-`--out` run on a fragment layout to `/contract.bundled.fluid.yaml`, and for `--format tgz` wrote the gzip tarball under that `.yaml` name, which `fluid validate` and `fluid plan` could not read as a contract. If an earlier `fluid ship` left that file beside your contract, delete it. + +### When to use the macro vs. the individual stages + +| Use `fluid ship` when | Use individual stages when | +|---|---| +| You've already iterated and want a clean push | You're debugging a failing stage | +| For demo and quickstart speed | You need stage-specific flags `ship` doesn't expose (per-stage `--out` paths, `bundle --format`, `plan --html`) | +| | You want to inspect `plan.json` before apply, or bind the apply to the plan | + +In 0.30.1 and earlier, for the production-grade 11-stage pipeline (cryptographic plan binding, drift gating, supply-chain signing), use [`fluid generate ci`](./generate.md#fluid-generate-ci) to emit the full pipeline for your CI system instead of `fluid ship`. *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* The pipeline `fluid generate ci` writes calls `fluid ship` for every stage, so the macro and the generated pipeline run the same code: see [What a generated pipeline runs](../advanced/operating-in-ci.md#what-a-generated-pipeline-runs). + +### Exit codes of the four-stage macro + +`fluid ship` relays the exit code of the **first failing stage**: + +| Code | Meaning | +|---|---| +| `0` | All stages passed | +| Non-zero | Exit code of the failing stage (e.g. `1` from `validate` for a schema error) | + +With no contract path and no `contract.fluid.yaml` in the current directory, `ship` exits `1` with `contract_required`. Stage logs are written to stdout and stderr unmodified: the macro doesn't buffer or reformat them. Pipe to `tee` or to your CI system's log capture as usual. + ## See also - [`fluid validate`](./validate.md), [`fluid bundle`](./bundle.md), [`fluid plan`](./plan.md), [`fluid apply`](./apply.md): the stages `ship` chains From b8f25864c7cbbe20022dacfcd4f5f1c7fa744411 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:31:43 +0100 Subject: [PATCH 7/7] docs: provider baselines say when CLI 0.18.1 was released and what the site is pinned to --- docs/providers/aws.md | 2 +- docs/providers/gcp.md | 2 +- docs/providers/local.md | 2 +- docs/providers/snowflake.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/providers/aws.md b/docs/providers/aws.md index b0c2c4b..045887c 100644 --- a/docs/providers/aws.md +++ b/docs/providers/aws.md @@ -2,7 +2,7 @@ Deploy data products to Amazon Web Services (S3, Glue, Athena, Lake Formation) with the same contract and CLI commands you use for the other providers. -**Docs Baseline:** CLI `0.18.1`. A part marked *(since …)* or *(on forge-cli `main`)* describes a later change.
+**Docs Baseline:** CLI `0.18.1`, released on 5 October 2026. This site is pinned to CLI `0.35.0`, and a part marked *(since …)* or *(on forge-cli `main`)* describes a later change.
**`fluid apply` creates:** S3 buckets, Glue databases and tables, Lake Formation registrations and grants, KMS keys, S3 lifecycle rules
**`fluid verify` reads:** Glue, Athena, S3, KMS, Lake Formation diff --git a/docs/providers/gcp.md b/docs/providers/gcp.md index 6ccd4a2..1d9274d 100644 --- a/docs/providers/gcp.md +++ b/docs/providers/gcp.md @@ -1,7 +1,7 @@ # GCP Provider **Status:** ✅ Production Ready -**Docs Baseline:** CLI `0.18.1`. A part marked *(since …)* or *(on forge-cli `main`)* describes a later change.
+**Docs Baseline:** CLI `0.18.1`, released on 5 October 2026. This site is pinned to CLI `0.35.0`, and a part marked *(since …)* or *(on forge-cli `main`)* describes a later change.
**Services:** BigQuery, Cloud Storage, Pub/Sub, IAM, Data Catalog policy tags, Cloud KMS > **Why it matters** diff --git a/docs/providers/local.md b/docs/providers/local.md index 23e08e0..4f50d00 100644 --- a/docs/providers/local.md +++ b/docs/providers/local.md @@ -1,7 +1,7 @@ # Local Provider **Status:** ✅ Production Ready -**Docs Baseline:** CLI `0.18.1`
+**Docs Baseline:** CLI `0.18.1`, released on 5 October 2026. This site is pinned to CLI `0.35.0`, and a part marked *(since …)* or *(on forge-cli `main`)* describes a later change.
**Engine:** DuckDB (1.5.0 or later) > **Why it matters** diff --git a/docs/providers/snowflake.md b/docs/providers/snowflake.md index 79425a4..45aa185 100644 --- a/docs/providers/snowflake.md +++ b/docs/providers/snowflake.md @@ -3,7 +3,7 @@ Deploy data products to Snowflake Data Cloud (databases, schemas and tables) using the same contract format and CLI commands as the other providers. **Status:** ✅ Production -**Docs Baseline:** CLI `0.18.1`
+**Docs Baseline:** CLI `0.18.1`, released on 5 October 2026. This site is pinned to CLI `0.35.0`, and a part marked *(since …)* or *(on forge-cli `main`)* describes a later change.
**Tested Services:** Databases, Schemas, Tables > **Why it matters**