Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The site is organised the way a reader uses it (get started, tutorials, how-to g

## Current Versioning

- Current CLI release documented here: `0.33.0`
- Current CLI release documented here: `0.34.0`
- Contract schema: `fluidVersion: 0.7.6` is the stable default; `0.7.7` is an opt-in preview

Those are different on purpose. `fluid version` reports the installed CLI release, while `fluidVersion` inside a contract selects the contract schema version.
Expand Down
4 changes: 2 additions & 2 deletions docs/.vuepress/cli-version.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"supportedCliVersion": "0.33.0",
"supportedCliInstallSpec": "data-product-forge==0.33.0",
"supportedCliVersion": "0.34.0",
"supportedCliInstallSpec": "data-product-forge==0.34.0",
"supportedFluidContractVersion": "0.7.6",
"quickstartScaffoldVersion": "0.7.5",
"supportedSdkVersion": "0.10.0",
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Hello from the my-first-product data product,2026-10-05 10:35:49.479853+02

[Getting Started](./getting-started/README.md) walks through each step with its output, then makes the product read a CSV and adds a data-quality rule. It explains why it uses the `fluid.starter` blueprint rather than `fluid init --quickstart`.

This site documents CLI `0.33.0`, with contract schema `0.7.6` as the stable default and `0.7.7` as an [opt-in preview](./reference/preview-fields.md). `fluid version` is the CLI release you installed; `fluidVersion` inside a contract is the [schema version](./reference/README.md) it is written against. Which version each scaffold writes is in [Understand the version numbers](./getting-started/README.md#understand-the-version-numbers).
This site documents CLI `0.34.0`, with contract schema `0.7.6` as the stable default and `0.7.7` as an [opt-in preview](./reference/preview-fields.md). `fluid version` is the CLI release you installed; `fluidVersion` inside a contract is the [schema version](./reference/README.md) it is written against. Which version each scaffold writes is in [Understand the version numbers](./getting-started/README.md#understand-the-version-numbers).

## Follow one product from laptop to production

Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/environment-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ The CLI itself does not read these; the generated pipeline and the generated DAG

| Variable | Read by | Effect |
|---|---|---|
| `FLUID_PACKAGE_SPEC` | the generated Jenkins stage 0 | Package spec installed into the workspace venv. Defaults to the generating CLI version with the extras the contract needs, for example `data-product-forge[local]==0.33.0`. |
| `FLUID_PACKAGE_SPEC` | the generated Jenkins stage 0 | Package spec installed into the workspace venv. Defaults to the generating CLI version with the extras the contract needs, for example `data-product-forge[local]==0.34.0`. |
| `FLUID_PIP_INDEX_URL`, `FLUID_PIP_EXTRA_INDEX_URL` | the generated install step | Primary and extra pip index. Blank means PyPI. pip takes the highest version across both, so for private packages use one mirror that proxies PyPI in `FLUID_PIP_INDEX_URL` and leave the extra index empty; see [Operating in CI](./operating-in-ci.md#install). |
| `FLUID_ALLOW_PRERELEASE` | the generated install step | `true` adds `pip --pre`. |
| `FLUID_CONFIG_PATH` | set by generated pipelines | `./fluid_config`. Nothing in the CLI reads it. |
Expand Down
4 changes: 2 additions & 2 deletions docs/advanced/operating-in-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ fluid generate ci contract.fluid.yaml --system jenkins --out Jenkinsfile
[install-mode: pypi] Jenkinsfile written -> Jenkinsfile
|- Jenkins installs FLUID_PACKAGE_SPEC into $WORKSPACE/.fluid-venv (stage 0)
...
|- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.33.0
|- FLUID_PACKAGE_SPEC default: data-product-forge[local]==0.34.0
|- Jenkins plugins required: workflow-aggregator, git
```

Expand Down Expand Up @@ -172,7 +172,7 @@ Jenkins learns a job's parameters from its first build. As measured on a Jenkins

### Install

Stage 0 creates `$WORKSPACE/.fluid-venv`, because agents that follow PEP 668 refuse a bare `pip install`, and installs `FLUID_PACKAGE_SPEC` into it. The default is the version that generated the file with the extras the contract's base and overlays need, for example `data-product-forge[local]==0.33.0`. Every stage then runs `fluid` from that venv, and stage 0 fails if the first `fluid` on `PATH` is another one. The agent needs `python3` with the `venv` module, and `rsync` for a `file://` or `ssh://` stage 11 destination. The workspace is removed with Jenkins's core `deleteDir()`, so the `ws-cleanup` plugin is not needed.
Stage 0 creates `$WORKSPACE/.fluid-venv`, because agents that follow PEP 668 refuse a bare `pip install`, and installs `FLUID_PACKAGE_SPEC` into it. The default is the version that generated the file with the extras the contract's base and overlays need, for example `data-product-forge[local]==0.34.0`. Every stage then runs `fluid` from that venv, and stage 0 fails if the first `fluid` on `PATH` is another one. The agent needs `python3` with the `venv` module, and `rsync` for a `file://` or `ssh://` stage 11 destination. The workspace is removed with Jenkins's core `deleteDir()`, so the `ws-cleanup` plugin is not needed.

Pin a different version at build time with `FLUID_PACKAGE_SPEC=data-product-forge==X.Y.Z`. `FLUID_PIP_INDEX_URL` (blank means PyPI), `FLUID_PIP_EXTRA_INDEX_URL` and `FLUID_ALLOW_PRERELEASE` (`true` adds `pip --pre`) cover TestPyPI pilots and private mirrors. With `--install-mode dev-source` the file installs from a `/forge-cli-src` bind mount and fails if the mount is missing; it never falls back to PyPI.

Expand Down
2 changes: 1 addition & 1 deletion docs/cli/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# CLI Reference

This section tracks the command surface of `forge-cli` `0.33.0`, the release `pip install data-product-forge` installs. `fluid --help` shows the promoted commands.
This section tracks the command surface of `forge-cli` `0.34.0`, the release `pip install data-product-forge` installs. `fluid --help` shows the promoted commands.

::: tip Looking up *how to do something* (not *what a command does*)?
The page you actually want is [**CLI by task →**](./tasks/) — narrative walkthroughs organized around things you might be trying to accomplish, like "deploy to a new cloud", "add quality rules", "debug a failed run", or "add agent governance". Each task page links to the relevant commands. **This** page is the command index, grouped by purpose — better for lookup, worse for onboarding.
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/contract-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ The contract is loaded the way `fluid plan` loads it, with `$ref` fragments and
```yaml
- name: Schema gate
run: |
pip install data-product-forge==0.33.0
pip install data-product-forge==0.34.0
fluid contract-tests contract.fluid.yaml --baseline baseline.schema.json
```

Expand Down
4 changes: 2 additions & 2 deletions docs/cli/generate.md
Original file line number Diff line number Diff line change
Expand Up @@ -528,7 +528,7 @@ Options:
| `--no-generate-artifacts` | Skip the transformation and schedule stages, for reference-only contracts. Through 0.32.0 it was also set automatically when any build's `pattern` was `hybrid-reference`, `reference` or `external-reference`. It sets stage 3's default to off on every system. *(since 0.33.0, [forge-cli #1000](https://github.com/Agenticstiger/forge-cli/pull/1000))* Nothing sets it automatically. A contract with reference builds, or only reference builds, still runs stage 3. |
| `--install-mode {pypi,dev-source}` | 0.30.1: Jenkins and Tekton. `pypi` (default): the install step installs the package spec (Jenkins: `FLUID_PACKAGE_SPEC` into a virtual environment in the workspace). `dev-source` installs from a `/forge-cli-src` bind mount for contributor labs and fails if the mount is missing; on Tekton it also drops the `fluid-package-spec` and pip index parameters from `pipeline.yaml`. *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* Every system reads it. *(since 0.33.0, [forge-cli #1000](https://github.com/Agenticstiger/forge-cli/pull/1000))* `dev-source` runs `pip install -e /forge-cli-src`, which puts `fluid` on `PATH`, instead of exporting `PYTHONPATH`. On the six other systems it stops with a `FATAL` line when `/forge-cli-src/pyproject.toml` is missing, and when the install fails. The runner needs a package index, or `setuptools` and `setuptools-scm` installed with `PIP_NO_BUILD_ISOLATION=0` (pip reads it inverted). Jenkins keeps the agent's installed `fluid` with `PYTHONPATH=/forge-cli-src`. In 0.32.0 every other system exported `PYTHONPATH=/forge-cli-src` in the install step, which ended with the step and provided no `fluid` command, so the install step failed at `fluid --version`. |
| `--image IMAGE` | *(since 0.33.0, [forge-cli #1000](https://github.com/Agenticstiger/forge-cli/pull/1000))* Tekton only: the image the Task steps run. Default `python:3.12-slim`, pinned by digest. It takes `[registry[:port]/]name[:tag][@sha256:<digest>]`, and a value that is not an image reference is refused with `generate_ci_invalid_option`. Pin yours by digest too (`name:tag@sha256:<digest>`): a tag can move. Another system refuses the flag with `generate_ci_image_unsupported`. |
| `--fluid-package-spec SPEC` | Default of the `FLUID_PACKAGE_SPEC` parameter. Default: the version that generated the file, with the extras the contract's bindings need across its base and every overlay, such as `data-product-forge[local]==0.33.0` or `data-product-forge[aws,local]==0.33.0`. |
| `--fluid-package-spec SPEC` | Default of the `FLUID_PACKAGE_SPEC` parameter. Default: the version that generated the file, with the extras the contract's bindings need across its base and every overlay, such as `data-product-forge[local]==0.34.0` or `data-product-forge[aws,local]==0.34.0`. |
| `--fluid-env-default ENV` | Default of `FLUID_ENV`, the overlay the stages pass to `--env`, and the fallback the stages read, so a build with no parameters runs in `ENV`. Letters, digits, `.`, `_` and `-`, starting with a letter. Default `dev`. |
| `--apply-mode-default MODE` | Default of `APPLY_MODE`, the mode stage 6 plans and stage 7 applies: `dry-run`, `create-only`, `amend`, `amend-and-build`, `replace` or `replace-and-build`. Default `dry-run` for Jenkins, `amend` for Tekton. |
| `--default-publish-target TARGET` | Default of `PUBLISH_TARGETS`, and the fallback stage 10 reads when a build has no parameters. Default `datamesh-manager`. Pick the catalog your team publishes to, for example `fluid-command-center`, `horizon`, `datahub` or `collibra`. |
Expand Down Expand Up @@ -566,7 +566,7 @@ Below is the 0.30.1 measurement. As measured on 0.18.1 by generating each system

*(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* Every stage runs `fluid ship "$CONTRACT" --stage N`, and the header of the file says: "A change request (pull request build) stops after stage 6. A branch build applies after a person approves (APPLY_APPROVAL, an input step), one build per environment at a time (lockable-resources)." Stages 7 to 11 sit behind an `Approve apply` stage with an `input` step and a `milestone`, and then run under a `lock` named `fluid-apply-<FLUID_ENV>`. They run on the default branch only (`main`, or `--default-branch`), or in a job that builds no branch. `APPLY_APPROVAL` defaults to `true`: set it to `false` for an unattended job. A change request build has no applied-plan baseline for stage 5, because its own job never applies. The sections below describe the 0.30.1 file.

- **Install.** Stage 0 creates `$WORKSPACE/.fluid-venv` and installs `FLUID_PACKAGE_SPEC` into it, because an agent whose system Python is externally managed (PEP 668) refuses a bare `pip install`. The spec defaults to the version that generated the file, so a Jenkinsfile generated by 0.33.0 installs `data-product-forge[local]==0.33.0` and does not move when a new release ships. Override it per build with the `FLUID_PACKAGE_SPEC` parameter, `FLUID_PIP_INDEX_URL` and `FLUID_PIP_EXTRA_INDEX_URL` (TestPyPI), or `FLUID_ALLOW_PRERELEASE`. pip takes the highest version across every index it is given, so do not pair a test or private index with PyPI; see [Operating in CI](../advanced/operating-in-ci.md#install).
- **Install.** Stage 0 creates `$WORKSPACE/.fluid-venv` and installs `FLUID_PACKAGE_SPEC` into it, because an agent whose system Python is externally managed (PEP 668) refuses a bare `pip install`. The spec defaults to the version that generated the file, so a Jenkinsfile generated by 0.34.0 installs `data-product-forge[local]==0.34.0` and does not move when a new release ships. Override it per build with the `FLUID_PACKAGE_SPEC` parameter, `FLUID_PIP_INDEX_URL` and `FLUID_PIP_EXTRA_INDEX_URL` (TestPyPI), or `FLUID_ALLOW_PRERELEASE`. pip takes the highest version across every index it is given, so do not pair a test or private index with PyPI; see [Operating in CI](../advanced/operating-in-ci.md#install).
- **Plugins.** `workflow-aggregator` and `git`, plus `copyartifact` with `--diff-last-applied`. *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* Also `lockable-resources` and `junit`. The workspace is removed with the core `deleteDir()` step, so the `ws-cleanup` plugin is not needed.
- **The first build.** Jenkins learns a Pipeline's parameters from its first build, so a `buildWithParameters` request to a job that has never run is refused (observed on a Jenkins controller on 4 Oct 2026). Start the job once from the UI with no parameters. The generated file is written for that run: every parameter default is also the fallback each stage reads, so a first build, or the first build after a restart that re-seeds jobs from job-dsl or JCasC, runs what the declarations say. `--fluid-env-default` and `--apply-mode-default` set what that run does.
- **Stage 10.** `fluid publish` receives `--env "${FLUID_ENV:-dev}"` by default, so the catalog gets the contract with the overlay stages 5 to 9 used. Before 0.16.3 `fluid publish` had no `--env`, and the default was off. Pass `--no-publish-include-env` only when the pipeline installs a CLI older than `fluid publish --env`. *(since 0.31.0, [forge-cli #942](https://github.com/Agenticstiger/forge-cli/pull/942))* The flag is removed, and ship always passes `--env`. GitLab CI, GitHub Actions, Azure DevOps and Bitbucket pass `--env "${FLUID_ENV:-dev}"` to `fluid publish` in their default output; the default CircleCI output has no `fluid publish` step.
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/scaffold-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ The generated file installs `data-product-forge` from PyPI (`pip install --quiet
| publish | On `main` with `DMM_API_URL` set: `fluid publish $CONTRACT --catalog $CATALOG`. |

::: warning Pin the version in a pipeline that holds credentials
An unpinned `pip install data-product-forge` takes whatever release is latest when the job runs, and the same job holds the credentials the apply stage uses. After generating, edit the install line to pin the version you tested, for example `pip install --quiet "data-product-forge==0.33.0"`, and move the pin deliberately when you upgrade. [`fluid generate ci`](./generate.md#fluid-generate-ci) pins the install to the version that generated the file.
An unpinned `pip install data-product-forge` takes whatever release is latest when the job runs, and the same job holds the credentials the apply stage uses. After generating, edit the install line to pin the version you tested, for example `pip install --quiet "data-product-forge==0.34.0"`, and move the pin deliberately when you upgrade. [`fluid generate ci`](./generate.md#fluid-generate-ci) pins the install to the version that generated the file.
:::

The settings are `CONTRACT` (the path you passed), `PROVIDER` (default `default`), `BUILD_ID` (default empty; set it to a `builds[].id` to run that build), `CATALOG` (default `datamesh-manager`), `AIRFLOW_DAGS_DEST` and `DMM_API_URL`. GitLab reads them as CI/CD variables, GitHub Actions as `env`, repository `vars` and `secrets`, Jenkins from the `environment` block.
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/version.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Python: 3.11.15 (darwin)
System: Darwin arm64
```

`fluid --version` is the top-level flag and prints one line, `FLUID Forge CLI v0.33.0`.
`fluid --version` is the top-level flag and prints one line, `FLUID Forge CLI v0.34.0`.

## Options

Expand Down
2 changes: 1 addition & 1 deletion docs/faq/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ pip install --upgrade "data-product-forge[local]"
fluid version
```

Then follow the [upgrade guide](../upgrading.md). It has a checklist for each version you cross, because some upgrades need action: `0.17.0`, for example, moves remote state to a per-provider key for a contract that uses a per-contract state key, and needs one `fluid schedule-sync` to retire old Airflow DAGs if you sync Airflow DAGs, and `0.16.3` changed some results on purpose. Pin an exact version in CI (`data-product-forge==0.33.0`), not a range.
Then follow the [upgrade guide](../upgrading.md). It has a checklist for each version you cross, because some upgrades need action: `0.17.0`, for example, moves remote state to a per-provider key for a contract that uses a per-contract state key, and needs one `fluid schedule-sync` to retire old Airflow DAGs if you sync Airflow DAGs, and `0.16.3` changed some results on purpose. Pin an exact version in CI (`data-product-forge==0.34.0`), not a range.

Your contracts do not need a new `fluidVersion`: `0.27.0` reads `0.7.1` to `0.7.6`, and `0.7.7` as a preview.

Expand Down
4 changes: 2 additions & 2 deletions docs/getting-started/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ fluid version
```text
╭───────────────────────────────────── 📦 Version Information ─────────────────────────────────────╮
│ FLUID CLI │
│ Version: 0.33.0 │
│ Version: 0.34.0 │
│ API: v1 │
│ │
│ Supported Specifications: │
Expand Down Expand Up @@ -55,7 +55,7 @@ The table lists the extras a first project needs. A base install has none of `li

```bash
pip install "data-product-forge[local,gcp]" # several extras at once
pip install "data-product-forge[local]==0.33.0" # an exact version, for CI
pip install "data-product-forge[local]==0.34.0" # an exact version, for CI
```

A cloud `fluid apply` runs OpenTofu, so it also needs `tofu` 1.6.0 or newer on your `PATH`, or `fluid apply --ensure-opentofu`, which downloads a pinned build from the OpenTofu release and checks it against that release's `SHA256SUMS` (integrity, not a signature). *(since 0.31.0, [forge-cli #899](https://github.com/Agenticstiger/forge-cli/pull/899))* The minimum is 1.12.0, and the pinned build is 1.13.9. For production CI, install OpenTofu with a cosign- or gpg-verified install instead, and keep `tofu` on the runner image. The local path in this tutorial does not use OpenTofu.
Expand Down
2 changes: 1 addition & 1 deletion docs/providers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Fluid Forge uses one contract format across local and provider-backed execution

## Docs baseline

- CLI release covered by the primary docs: `0.33.0`
- CLI release covered by the primary docs: `0.34.0`
- Contract schema: `0.7.6` is the current stable version, and the default since `0.26.0`; `0.7.7` is a preview you opt into with `fluidVersion: "0.7.7"`. The GCP retention, encryption and `binding.principals` fields need `0.7.6`.
- Which `fluidVersion` each scaffolding command writes is listed in [`fluid init`](../cli/init.md#which-fluidversion-each-path-writes).

Expand Down
4 changes: 2 additions & 2 deletions docs/recipes/per-environment-overlays.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,14 +249,14 @@ jobs:
matrix:
env: [dev, staging, prod]
steps:
- run: pip install "data-product-forge==0.33.0"
- run: pip install "data-product-forge==0.34.0"
- run: fluid validate contract.fluid.yaml --env ${{ matrix.env }}
- run: fluid plan contract.fluid.yaml --env ${{ matrix.env }}
```

Pin the version: an unpinned `pip install` in a pipeline takes whatever is
newest on the day. Add the extras your clouds need, for example
`"data-product-forge[gcp]==0.33.0"`.
`"data-product-forge[gcp]==0.34.0"`.

## See also

Expand Down
Loading
Loading