Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 28 additions & 6 deletions docs/releases/0.7.7.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,16 @@
0.7.7 is a **preview**. A contract uses it only by declaring `fluidVersion: "0.7.7"`, and its fields and their meaning can still change before it is promoted to stable. The latest stable version is [0.7.6](/fluid/releases/0.7.6). See [Stable and preview versions](/fluid/schema/versions#stable-and-preview-versions).
:::

0.7.7 lets a column declare nested data (structs, arrays and maps, to any depth) and lets an embedded-logic build describe the Kafka records it reads and how they become rows, in place of a SQL statement. Since its first publication here it has also gained rules about valid data (invariants written in FLUID SQL/1), identifier formats, placeholder values and shared attributes, named relationships and the validity window of a slowly changing dimension, a schema registry definition, Spark builds, the `lakeflow` engine, and a stricter grammar for secret references.
0.7.7 lets a column declare nested data (structs, arrays and maps, to any depth) and lets an embedded-logic build describe the Kafka records it reads and how they become rows, in place of a SQL statement. Since its first publication here it has also gained rules about valid data (invariants written in FLUID SQL/1), identifier formats, placeholder values and shared attributes, named relationships and the validity window of a slowly changing dimension, a schema registry definition, Spark builds, the `lakeflow` engine, a stricter grammar for secret references, and quoted levels in an `enum` column type.

- **Schema:** [`fluid-schema-0.7.7.json`](/fluid/schema/fluid-schema-0.7.7.json) · **Reference:** [`specs/0.7.7/fluid-spec.html`](/fluid/specs/0.7.7/fluid-spec.html) · **Diff:** [`diff-0.7.6-to-0.7.7.md`](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.6-to-0.7.7.md)
- **Compatibility:** additive over 0.7.6 except for one narrowing, in secret references (see [Secret references](#secret-references)). A valid 0.7.6 contract stays valid when it declares `"0.7.7"` unless it names a `vault://` secret without a `#field`. `scripts/check-compat.py --from 0.7.6 --to 0.7.7` reports that one case, and it is recorded as a waiver in `scripts/compat-waivers.txt`.
- **Compatibility:** additive over 0.7.6 except for two narrowings, in secret references (see [Secret references](#secret-references)) and at the edge of the `enum` column type (see [Enum levels](#enum-levels)). A valid 0.7.6 contract stays valid when it declares `"0.7.7"` unless it names a `vault://` secret without a `#field`, or an `enum(...)` type with an empty item, a quote that does not open and close a level, or a tab, a newline or another whitespace control character inside a quoted level. `scripts/check-compat.py --from 0.7.6 --to 0.7.7` reports both, and each is recorded as a waiver in `scripts/compat-waivers.txt`.

---

## What is new

Every addition below is optional. These are the property paths present in 0.7.7 and absent in 0.7.6. The text of each comes from the schema's own descriptions, which are the place to read the rules in full.
Every addition below is optional. These are the property paths and values present in 0.7.7 and absent in 0.7.6. The text of each comes from the schema's own descriptions, which are the place to read the rules in full.

| Path | What it declares |
|---|---|
Expand All @@ -32,20 +32,42 @@ Every addition below is optional. These are the property paths present in 0.7.7
| `exposes[].semantics.relationships[]`, with `sharedColumns` | Named joins from one expose to another, composite keys included, and columns copied through a relationship. |
| `exposes[].semantics.dimensions[].typeParams.validityParams` | Marks a time dimension as one bound of the validity window of a type II slowly changing dimension. The entity type `natural` is described as that dimension's key. |
| `secretRef` (shared `$defs/secretRef`), `connection.password`, `auth.token` | A per-scheme grammar for secret references, and two write-only members that name what a contract may not hold. |
| `schema[].type` as `enum(...)`, and the `type` of `element`, `key` and `value` | The levels of an enum column, as words or as quoted strings that may hold any character but a control character. See [Enum levels](#enum-levels). |

`fluidVersion` accepts `"0.7.7"` in addition to `"0.7.3"` to `"0.7.6"`. `build.engine` gains `lakeflow`.

One existing rule changes shape. Through 0.7.6, an embedded-logic build's `properties` always required `sql`. In 0.7.7 it requires `sql` unless the build has a `stream` and either its `parameters.flink` declares `management`, or it has a `mapping` and a `parameters.spark`; the build is then described by the stream and the mapping alone. A `mapping` without a `stream` is refused. Every document that satisfied the old rule satisfies the new one.

The schema carries a second change of meaning, in `secretRef`. See the next section.
The schema carries two further changes of meaning, in `secretRef` and in the `enum` column type. See the next two sections.

---

## Secret references

`secretRef` is now one shared definition, `$defs/secretRef`, used by an acquisition source's `connection`, by the `auth` of an acquisition build's `deployment`, and by the `auth` of the `kafka-connect` `schema_registry`. It names a secret held in a store and never carries it. The grammar is `<scheme>://<locator>[#field][?version=n]`. The schemes are `env`, `file`, `vault`, `aws-sm`, `gcp-sm` and `azure-kv`, with the aliases `openbao` (vault), `aws`, `gcp` and `azure`. Any other lower-case scheme, such as `op://`, is passed to the pack that resolves it. `#field` picks one key of a secret that holds several, and `?version=` pins a version where the store keeps them. Each scheme has its own pattern in the schema.

Through 0.7.6 and in the first copy of 0.7.7 published here, a `secretRef` was any string of the form `<scheme>://<anything>`. That is the one place 0.7.7 accepts less than 0.7.6: a `vault://` reference now needs a `#field`, so `vault://secret/data/crm-db` is refused and `vault://secret/data/crm-db#password` is valid. A reference with another scheme or with a `#field` is unaffected. `connection.password` and `auth.token` are declared `writeOnly`, with the description that a contract may not hold the value and names the secret with `secretRef` instead.
Through 0.7.6 and in the first copy of 0.7.7 published here, a `secretRef` was any string of the form `<scheme>://<anything>`. That is the first place where 0.7.7 accepts less than 0.7.6: a `vault://` reference now needs a `#field`, so `vault://secret/data/crm-db` is refused and `vault://secret/data/crm-db#password` is valid. A reference with another scheme or with a `#field` is unaffected. `connection.password` and `auth.token` are declared `writeOnly`, with the description that a contract may not hold the value and names the secret with `secretRef` instead.

---

## Enum levels

An enum column names its levels in the parameters of its `type`, comma-separated. A level is a list of words, as before (`enum(IE, United Kingdom)`), or a quoted string in single or double quotes. A quoted level may hold any character but a control character, and writes its own quote twice, as a SQL string literal does: `enum('Dr.', '>10000', 'O''Brien', 'a, b')`. The `type` of a nested `element`, `key` or `value` is the same definition.

Through 0.7.6, and in the copies of 0.7.7 published here before forge-cli v0.32.0, the parameters of every type were any run of letters, digits, underscores, spaces, commas, quotes and hyphens, so a level such as `Dr.` or `>10000` could not be written. Every type other than `enum` keeps that character set.

That is the second place where 0.7.7 accepts less than 0.7.6, and it is narrow. The parameters of an enum must now read as a list of levels, so 0.7.7 refuses an empty item, a quote that does not open and close a level, and a tab, a newline or another whitespace control character inside a quoted level, all of which 0.7.6 accepted. A valid 0.7.6 contract with none of these stays valid, and `enum`, `enum()` and `enum(IE, United Kingdom)` read as before.

| `type` | 0.7.6 | 0.7.7 |
|---|---|---|
| `enum(IE, United Kingdom)` | valid | valid |
| `enum('Dr.', '>10000', 'O''Brien', 'a, b')` | refused | valid |
| `enum("Dr.", "Cir.")` | refused | valid |
| `enum(a,,b)` | valid | refused |
| `enum('it's')` | valid | refused |
| `enum('a' 'b')` | valid | refused |

`scripts/compat-waivers.txt` records the comparison of the two patterns that settled this, on more than a million strings.

---

Expand Down Expand Up @@ -184,7 +206,7 @@ The schema asks only for a text. `fluid validate` in the reference implementatio

## A complete contract

The reference implementation can deploy a managed Flink job for a build like this one: a Kubernetes cluster that runs the Flink Kubernetes Operator, writing an Iceberg REST catalog table. That is behaviour of one implementation (non-normative), and it needs operator-side allow-lists for the cluster context, namespace, image, service account and secrets the contract names. The contract below validates against the 0.7.7 JSON Schema (Draft 2020-12) and with `data-product-forge` 0.31.0 (`fluid validate` prints `✅ Valid FLUID contract (schema v0.7.7)`). `fluid validate` refuses a managed Flink build that has no build `id` or no Iceberg output named in `outputs`, so the contract carries both.
The reference implementation can deploy a managed Flink job for a build like this one: a Kubernetes cluster that runs the Flink Kubernetes Operator, writing an Iceberg REST catalog table. That is behaviour of one implementation (non-normative), and it needs operator-side allow-lists for the cluster context, namespace, image, service account and secrets the contract names. The contract below validates against the 0.7.7 JSON Schema (Draft 2020-12) and with `data-product-forge` 0.32.0 (`fluid validate` prints `✅ Valid FLUID contract (schema v0.7.7)`). `fluid validate` refuses a managed Flink build that has no build `id` or no Iceberg output named in `outputs`, so the contract carries both.

```yaml
fluidVersion: "0.7.7"
Expand Down
4 changes: 2 additions & 2 deletions docs/releases/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Release notes for the FLUID specification, newest first. Each entry links to its

| Version | Status | Codename | Compatibility | Summary |
|---|---|---|---|---|
| **[0.7.7](/fluid/releases/0.7.7)** | **Preview** | Recursive Streaming Types | ⚠️ Additive over 0.7.6, except that a `vault://` secret reference needs a `#field` | Adds nested columns (`fields`, `element`, `key` and `value` on a column), and on an embedded-logic build the Kafka `stream` it reads and the `mapping` that turns its records into rows, a schema registry definition, Spark builds and the `lakeflow` engine. Adds rules about valid data (`invariants`), identifier formats, placeholders and shared attributes, and named relationships. `sql` is no longer required when a managed Flink or Spark build has a stream. |
| **[0.7.7](/fluid/releases/0.7.7)** | **Preview** | Recursive Streaming Types | ⚠️ Additive over 0.7.6, except that a `vault://` secret reference needs a `#field` and an `enum(...)` column type refuses an empty item, a quote that does not open and close a level, or a whitespace control character inside a quoted level | Adds nested columns (`fields`, `element`, `key` and `value` on a column), and on an embedded-logic build the Kafka `stream` it reads and the `mapping` that turns its records into rows, a schema registry definition, Spark builds and the `lakeflow` engine. Adds rules about valid data (`invariants`), identifier formats, placeholders and shared attributes, and named relationships. An `enum` column type can name quoted levels, such as `enum('Dr.', '>10000')`. `sql` is no longer required when a managed Flink or Spark build has a stream. |
| **[0.7.6](/fluid/releases/0.7.6)** | **Stable (latest)** | Declarative Packaging Modes | ✅ Additive over 0.7.5 | Adds top-level `packaging` and `consumers`, cross-mesh pins on `consumes[]` (`upstreamWorkspace` + `upstreamDigest`), `aggParams` on semantic measures, and on bindings `encryption.kms`, `principals` and Lake Formation `bucketPolicy`; an expose's `lifecycle` gains `expire`, masking rules gain typed `params`, and access policies gain `rowFilters` and `access: mask`. Builds can declare `engine: flink` and a `continuous` trigger, and an Iceberg sink can name a REST catalog's credentials by environment variable (`sink.catalogAuth`). Optional Bitol-shaped fields: `metadata.team`, an object root `description`, `consumes[].contractId` and `contractVersion`, `exposes[].contract.id`, column `quality`, `binding.location.host` and `binding.serialization`. |
| **[0.7.5](/fluid/releases/0.7.5)** | Stable | Streaming Kafka → Iceberg Sink & Confluent Tableflow | ✅ Additive over 0.7.4 | Adds an opt-in Iceberg streaming sink on the `kafka-connect` acquisition engine, the `confluent` (Tableflow) and `pgvector` binding platforms, the `pgvector_table` format and `binding.vectorConfig`, and location fields for Tableflow, Iceberg catalogs, Redshift Serverless and Kinesis. |
| **[0.7.4](/fluid/releases/0.7.4)** | Stable | Runtime agentPolicy Enforcement at the MCP Gateway | ✅ Additive over 0.7.3 | Adds the optional `exposes[].mcp` block, the `postgres` binding platform, and the `postgres_table` / `athena_table` / `glue_table` formats. |
Expand All @@ -18,7 +18,7 @@ Release notes for the FLUID specification, newest first. Each entry links to its

## Reading the compatibility column

"Additive" means what [`scripts/check-compat.py`](https://github.com/open-data-protocol/fluid/blob/main/scripts/check-compat.py) checks on every pull request: no document that validates under one version stops validating under the next. Every transition in the table is additive except **0.7.1 → 0.7.2**, where `$defs/notification` gained `additionalProperties: false`. That break is recorded in [`scripts/compat-waivers.txt`](https://github.com/open-data-protocol/fluid/blob/main/scripts/compat-waivers.txt) and explained in the [0.7.2 note](/fluid/releases/0.7.2#backward-compatibility-with-0-7-1-one-narrowing-change).
"Additive" means what [`scripts/check-compat.py`](https://github.com/open-data-protocol/fluid/blob/main/scripts/check-compat.py) checks on every pull request: no document that validates under one version stops validating under the next. Every stable transition in the table is additive except **0.7.1 → 0.7.2**, where `$defs/notification` gained `additionalProperties: false`. That break is recorded in [`scripts/compat-waivers.txt`](https://github.com/open-data-protocol/fluid/blob/main/scripts/compat-waivers.txt) and explained in the [0.7.2 note](/fluid/releases/0.7.2#backward-compatibility-with-0-7-1-one-narrowing-change). The 0.7.7 preview narrows in two small places, secret references and the `enum` column type. Both are recorded in the same file and explained in the [0.7.7 note](/fluid/releases/0.7.7).

Additive does **not** mean that an older `fluidVersion` may use newer fields. A document is validated against the schema of the version it declares, so to use a field you declare the version that added it. See [Choosing `fluidVersion`](/fluid/schema/versions#choosing-fluidversion).

Expand Down
4 changes: 3 additions & 1 deletion docs/schema/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Human-readable summaries of what changed between consecutive versions of the FLU

[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.6-to-0.7.7.md) · [Release note →](/fluid/releases/0.7.7)

One narrowing, in secret references (see **Changed**); everything else is additive. **Preview:** the 0.7.7 schema can still change before it is promoted. The file published here was re-vendored from forge-cli v0.31.0; it was first published from v0.27.0, and the items under "Added since the v0.27.0 copy" came after that.
Two narrowings, one in secret references and one at the edge of the `enum` column type (see **Changed**); everything else is additive. **Preview:** the 0.7.7 schema can still change before it is promoted. The file published here was re-vendored from forge-cli v0.32.0; it was first published from v0.27.0, and the items under "Added since the v0.27.0 copy" came after that.

**Added**

Expand All @@ -25,11 +25,13 @@ One narrowing, in secret references (see **Changed**); everything else is additi
- `build.engine` value `lakeflow`.
- `packaging.containers` keys `lfTagVocabulary`, `emrApplication`, `schemaRegistry`, `athenaWorkgroup`, `kmsKey` and `mwaaEnvironment`.
- `connection.password` and `auth.token` as `writeOnly` members, on an acquisition source and on an acquisition build's `deployment.auth`.
- Quoted levels in an `enum` column type, such as `enum('Dr.', '>10000', 'O''Brien', 'a, b')`: a level is a word list or a quoted string that may hold any character but a control character, with its own quote doubled as SQL writes it. The `type` of a nested `element`, `key` and `value` is the same definition, so it accepts them too.

**Changed:**

- `embeddedLogicPattern` no longer lists `sql` as unconditionally required. It requires `sql` unless the build has a `stream` and either `parameters.flink` declares `management`, or it has a `mapping` and `parameters.spark`; `mapping` requires `stream` (`dependentRequired`). Every document that carried `sql` is unaffected.
- `secretRef` (an acquisition source's `connection`, the `auth` of an acquisition build's `deployment`, and the `kafka-connect` `schema_registry.auth`) is now the shared `$defs/secretRef`: one pattern per scheme (`env`, `file`, `vault`, `aws-sm`, `gcp-sm`, `azure-kv`, and other lower-case schemes). It was any string of the form `<scheme>://<anything>`. This is the narrowing: a `vault://` reference now needs a `#field`, so `vault://secret/data/crm-db` is refused and `vault://secret/data/crm-db#password` is valid. `scripts/check-compat.py` reports it and `scripts/compat-waivers.txt` records it. See the [0.7.7 notes](/fluid/releases/0.7.7#secret-references).
- The parameters of an `enum` column `type` are now a comma-separated list of levels, where 0.7.6 accepted any run of letters, digits, underscores, spaces, commas, quotes and hyphens. This is the second narrowing, and a narrow one: 0.7.6 accepted an empty item (`enum(a,,b)`), a quote that does not open and close a level (`enum('it's')`, `enum('a' 'b')`) and a tab, a newline or another whitespace control character inside a quoted level, and 0.7.7 refuses each. Every other type keeps its characters, and `enum`, `enum()` and `enum(IE, United Kingdom)` are unaffected. `scripts/check-compat.py` reports the pattern as changed and `scripts/compat-waivers.txt` records the comparison that settled it. See the [0.7.7 notes](/fluid/releases/0.7.7#enum-levels).

## 0.7.5 to 0.7.6: Declarative Packaging Modes

Expand Down
Loading
Loading