Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 46 additions & 6 deletions docs/releases/0.7.7.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,26 +4,48 @@
0.7.7 is a **preview**. A contract uses it only by declaring `fluidVersion: "0.7.7"`, and its fields and their meaning can still change before it is promoted to stable. The latest stable version is [0.7.6](/fluid/releases/0.7.6). See [Stable and preview versions](/fluid/schema/versions#stable-and-preview-versions).
:::

0.7.7 lets a column declare nested data (structs, arrays and maps, to any depth) and lets an embedded-logic build describe the Kafka records it reads and how they become rows, in place of a SQL statement.
0.7.7 lets a column declare nested data (structs, arrays and maps, to any depth) and lets an embedded-logic build describe the Kafka records it reads and how they become rows, in place of a SQL statement. Since its first publication here it has also gained rules about valid data (invariants written in FLUID SQL/1), identifier formats, placeholder values and shared attributes, named relationships and the validity window of a slowly changing dimension, a schema registry definition, Spark builds, the `lakeflow` engine, and a stricter grammar for secret references.

- **Schema:** [`fluid-schema-0.7.7.json`](/fluid/schema/fluid-schema-0.7.7.json) · **Reference:** [`specs/0.7.7/fluid-spec.html`](/fluid/specs/0.7.7/fluid-spec.html) · **Diff:** [`diff-0.7.6-to-0.7.7.md`](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.6-to-0.7.7.md)
- **Compatibility:** additive over 0.7.6. `scripts/check-compat.py --from 0.7.6 --to 0.7.7` reports no narrowing change, so a valid 0.7.6 contract stays valid when it declares `"0.7.7"`.
- **Compatibility:** additive over 0.7.6 except for one narrowing, in secret references (see [Secret references](#secret-references)). A valid 0.7.6 contract stays valid when it declares `"0.7.7"` unless it names a `vault://` secret without a `#field`. `scripts/check-compat.py --from 0.7.6 --to 0.7.7` reports that one case, and it is recorded as a waiver in `scripts/compat-waivers.txt`.

---

## What is new

Every addition below is optional. These are the property paths present in 0.7.7 and absent in 0.7.6.
Every addition below is optional. These are the property paths present in 0.7.7 and absent in 0.7.6. The text of each comes from the schema's own descriptions, which are the place to read the rules in full.

| Path | What it declares |
|---|---|
| `schema[].fields`, `schema[].element`, `schema[].key`, `schema[].value` | The parts of a nested column: the child columns of a struct, the element type of an array, the key and value types of a map. They are available wherever a column is: `exposes[].contract.schema` and `stream.schema`. |
| `build(s).properties.stream` | The Kafka records a build reads: topic, value format, value fields, an optional key and Kafka metadata. |
| `build(s).properties.mapping` | How the stream's records become the output's rows: columns read from another path, renamed or cast, and an explode over an array. |
| `build(s).properties.stream.registry`, `binding.properties.registry` | A `schemaRegistry` definition: the registry that holds a topic's record schema, Glue or Confluent. |
| `build(s).properties.stream.trigger`, `.watermark`, `.checkpoint`, `.startingOffsets` | When the engine starts a micro-batch, the event-time watermark, where offsets and state are kept (an `s3://` URI), and where a job with no checkpoint starts reading. |
| `build(s).properties.mapping.columns[].maskAtIngest` | Opt-in `hash` of a mapped column before it lands. |
| `build(s).properties.parameters.spark` | Where and how a Spark build runs: `runtime` (`emr-serverless`, `emr-ec2`, `emr-eks` or `glue`), `mode`, the application or cluster, release, role, `sparkConf` and the Kafka `source`. |
| `build(s).engine` value `lakeflow` | A Databricks Lakeflow Declarative Pipeline: `properties.sql` is the `SELECT` of the dataset an expose binds. |
| `packaging.containers` keys `lfTagVocabulary`, `emrApplication`, `schemaRegistry`, `athenaWorkgroup`, `kmsKey`, `mwaaEnvironment` | Ownership of six AWS container kinds. `mwaaEnvironment` is shared only. |
| `exposes[].contract.invariants[]` | Rules about valid data: a `check`, a `derive` of a column (with `expr`, or with an `aggregate` over child rows) or a functional `dependency`, each with an optional `when` and a band of violating rows. |
| `schema[].identifierFormat`, `schema[].placeholders`, `schema[].sharedAttribute` | The identifier formats of a string column, its placeholder values, and the attribute it shares with other columns of the contract. |
| `schema[].protobuf` | The column's field number (`field`) when the topic's value is Protobuf. |
| `exposes[].semantics.relationships[]`, with `sharedColumns` | Named joins from one expose to another, composite keys included, and columns copied through a relationship. |
| `exposes[].semantics.dimensions[].typeParams.validityParams` | Marks a time dimension as one bound of the validity window of a type II slowly changing dimension. The entity type `natural` is described as that dimension's key. |
| `secretRef` (shared `$defs/secretRef`), `connection.password`, `auth.token` | A per-scheme grammar for secret references, and two write-only members that name what a contract may not hold. |

`fluidVersion` accepts `"0.7.7"` in addition to `"0.7.3"` to `"0.7.6"`. No enum gains a value.
`fluidVersion` accepts `"0.7.7"` in addition to `"0.7.3"` to `"0.7.6"`. `build.engine` gains `lakeflow`.

One existing rule changes shape. Through 0.7.6, an embedded-logic build's `properties` always required `sql`. In 0.7.7 it requires `sql` unless the build has a `stream` and its `parameters.flink` declares `management`; the build is then described by the stream and the mapping alone. A `mapping` without a `stream` is refused. Every document that satisfied the old rule satisfies the new one.
One existing rule changes shape. Through 0.7.6, an embedded-logic build's `properties` always required `sql`. In 0.7.7 it requires `sql` unless the build has a `stream` and either its `parameters.flink` declares `management`, or it has a `mapping` and a `parameters.spark`; the build is then described by the stream and the mapping alone. A `mapping` without a `stream` is refused. Every document that satisfied the old rule satisfies the new one.

The schema carries a second change of meaning, in `secretRef`. See the next section.

---

## Secret references

`secretRef` is now one shared definition, `$defs/secretRef`, used by an acquisition source's `connection`, by the `auth` of an acquisition build's `deployment`, and by the `auth` of the `kafka-connect` `schema_registry`. It names a secret held in a store and never carries it. The grammar is `<scheme>://<locator>[#field][?version=n]`. The schemes are `env`, `file`, `vault`, `aws-sm`, `gcp-sm` and `azure-kv`, with the aliases `openbao` (vault), `aws`, `gcp` and `azure`. Any other lower-case scheme, such as `op://`, is passed to the pack that resolves it. `#field` picks one key of a secret that holds several, and `?version=` pins a version where the store keeps them. Each scheme has its own pattern in the schema.

Through 0.7.6 and in the first copy of 0.7.7 published here, a `secretRef` was any string of the form `<scheme>://<anything>`. That is the one place 0.7.7 accepts less than 0.7.6: a `vault://` reference now needs a `#field`, so `vault://secret/data/crm-db` is refused and `vault://secret/data/crm-db#password` is valid. A reference with another scheme or with a `#field` is unaffected. `connection.password` and `auth.token` are declared `writeOnly`, with the description that a contract may not hold the value and names the secret with `secretRef` instead.

---

Expand Down Expand Up @@ -142,9 +164,27 @@ build:

---

## Rules about valid data

`exposes[].contract.invariants[]` declares what a valid row looks like, in FLUID SQL/1: a closed SQL expression subset with exact numbers and two-valued truth, at most 4,096 code points and 64 levels deep. An invariant has exactly one form: `check`, a predicate every covered row satisfies; `derive` with `expr`, a column's value from the row and the rows its links reach; `derive` with `aggregate`, a column's value as a summary over child rows (`function`, `column`, `from`, `via`, `filter`, `ifNone`); or `dependency`, where equal `from` values mean equal `to` values. `when` narrows the rows it covers, `severity` is `error`, `warning` or `info`, and one `mustBe*` operator with a `unit` of `rows` or `percent` bounds the violating rows. Without one, the invariant holds only with no violating row. SQL outside the subset belongs in `contract.quality[]`.

A string column can also declare `identifierFormat`, a list of alternatives, each a mask (`9` a digit, `A` an upper-case letter, `X` either) with optional `prefixes`; `placeholders`, the values that stand in for a real value not known yet, which are exempt from the column's own value checks; and `sharedAttribute`, the attribute it carries with every other column that names the same one. Alternatives of one format must not overlap. A relationship's `sharedColumns` lists columns copied through it, each in both exposes' schemas under one name and of one type.

The schema asks only for a text. `fluid validate` in the reference implementation parses it and type-checks it against the expose (non-normative).

---

## Schema registries and Spark builds

`$defs/schemaRegistry` is one definition of the registry that holds a topic's record schema: `kind` is `glue` (the AWS Glue Schema Registry, with IAM authentication and so no secret) or `confluent` (the Confluent Schema Registry, or one that serves its REST API), with `url`, `registryName`, `subject`, `version`, `compatibility`, `auth` and `rules`. It appears at `stream.registry` and at `binding.properties.registry`. Who owns the subject decides what happens to it: a stream a build reads belongs to its producer, so the subject is verified against the declared schema and never written, and a topic an expose produces has its subject registered with the declared compatibility. The older spellings, `kafka-connect`'s `schema_registry` and a Flink build's `parameters.flink.source.schemaRegistryUrl`, stay valid and read as this definition with `kind: confluent`.

`parameters.spark` describes where a Spark build runs, on an embedded-logic build with `engine: spark`. `runtime` is required: `emr-serverless`, `emr-ec2`, `emr-eks` or `glue`. The build declares what it computes with `stream` and `mapping`, or with `sql` as Spark SQL; there is no entry point for user PySpark. `execution` keeps when the build runs and what happens when it fails, and `parameters.spark` keeps where Spark runs. `maskAtIngest: hash` on a mapped column needs `parameters.spark.maskKeySecretArn`, the Secrets Manager secret that holds the HMAC key; the schema declares that member and the check that links them lives in the Spark compiler.

---

## A complete contract

The reference implementation can deploy a managed Flink job for a build like this one: a Kubernetes cluster that runs the Flink Kubernetes Operator, writing an Iceberg REST catalog table. That is behaviour of one implementation (non-normative), and it needs operator-side allow-lists for the cluster context, namespace, image, service account and secrets the contract names. The contract below validates against the 0.7.7 JSON Schema (Draft 2020-12) and with `data-product-forge` 0.27.0 (`fluid validate` prints `✅ Valid FLUID contract (schema v0.7.7)`). `fluid validate` refuses a managed Flink build that has no build `id` or no Iceberg output named in `outputs`, so the contract carries both.
The reference implementation can deploy a managed Flink job for a build like this one: a Kubernetes cluster that runs the Flink Kubernetes Operator, writing an Iceberg REST catalog table. That is behaviour of one implementation (non-normative), and it needs operator-side allow-lists for the cluster context, namespace, image, service account and secrets the contract names. The contract below validates against the 0.7.7 JSON Schema (Draft 2020-12) and with `data-product-forge` 0.31.0 (`fluid validate` prints `✅ Valid FLUID contract (schema v0.7.7)`). `fluid validate` refuses a managed Flink build that has no build `id` or no Iceberg output named in `outputs`, so the contract carries both.

```yaml
fluidVersion: "0.7.7"
Expand Down
2 changes: 1 addition & 1 deletion docs/releases/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Release notes for the FLUID specification, newest first. Each entry links to its

| Version | Status | Codename | Compatibility | Summary |
|---|---|---|---|---|
| **[0.7.7](/fluid/releases/0.7.7)** | **Preview** | Recursive Streaming Types | ✅ Additive over 0.7.6 | Adds nested columns (`fields`, `element`, `key` and `value` on a column), and on an embedded-logic build the Kafka `stream` it reads and the `mapping` that turns its records into rows. `sql` is no longer required when a managed Flink build has a stream. |
| **[0.7.7](/fluid/releases/0.7.7)** | **Preview** | Recursive Streaming Types | ⚠️ Additive over 0.7.6, except that a `vault://` secret reference needs a `#field` | Adds nested columns (`fields`, `element`, `key` and `value` on a column), and on an embedded-logic build the Kafka `stream` it reads and the `mapping` that turns its records into rows, a schema registry definition, Spark builds and the `lakeflow` engine. Adds rules about valid data (`invariants`), identifier formats, placeholders and shared attributes, and named relationships. `sql` is no longer required when a managed Flink or Spark build has a stream. |
| **[0.7.6](/fluid/releases/0.7.6)** | **Stable (latest)** | Declarative Packaging Modes | ✅ Additive over 0.7.5 | Adds top-level `packaging` and `consumers`, cross-mesh pins on `consumes[]` (`upstreamWorkspace` + `upstreamDigest`), `aggParams` on semantic measures, and on bindings `encryption.kms`, `principals` and Lake Formation `bucketPolicy`; an expose's `lifecycle` gains `expire`, masking rules gain typed `params`, and access policies gain `rowFilters` and `access: mask`. Builds can declare `engine: flink` and a `continuous` trigger, and an Iceberg sink can name a REST catalog's credentials by environment variable (`sink.catalogAuth`). Optional Bitol-shaped fields: `metadata.team`, an object root `description`, `consumes[].contractId` and `contractVersion`, `exposes[].contract.id`, column `quality`, `binding.location.host` and `binding.serialization`. |
| **[0.7.5](/fluid/releases/0.7.5)** | Stable | Streaming Kafka → Iceberg Sink & Confluent Tableflow | ✅ Additive over 0.7.4 | Adds an opt-in Iceberg streaming sink on the `kafka-connect` acquisition engine, the `confluent` (Tableflow) and `pgvector` binding platforms, the `pgvector_table` format and `binding.vectorConfig`, and location fields for Tableflow, Iceberg catalogs, Redshift Serverless and Kinesis. |
| **[0.7.4](/fluid/releases/0.7.4)** | Stable | Runtime agentPolicy Enforcement at the MCP Gateway | ✅ Additive over 0.7.3 | Adds the optional `exposes[].mcp` block, the `postgres` binding platform, and the `postgres_table` / `athena_table` / `glue_table` formats. |
Expand Down
18 changes: 15 additions & 3 deletions docs/schema/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,28 @@ Human-readable summaries of what changed between consecutive versions of the FLU

[Full diff →](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.6-to-0.7.7.md) · [Release note →](/fluid/releases/0.7.7)

No narrowing change. **Preview:** the 0.7.7 schema can still change before it is promoted.
One narrowing, in secret references (see **Changed**); everything else is additive. **Preview:** the 0.7.7 schema can still change before it is promoted. The file published here was re-vendored from forge-cli v0.31.0; it was first published from v0.27.0, and the items under "Added since the v0.27.0 copy" came after that.

**Added**

- Nested columns: `column.fields` (a non-empty list of columns), `column.element`, `column.key` and `column.value`, the last three each a new closed `nestedType` (`type` required; `required`, `fields`, `element`, `key` and `value` optional, nesting again).
- `build(s).properties.stream`, a new closed `streamSource` definition: `kind` (`kafka`), `topic`, `format` (`json` | `avro-confluent`) and `schema` required; `key` (`format` `raw` | `json` | `avro-confluent`, `schema`, `prefix`) and `metadata[]` (`name`, `from`: `topic` | `partition` | `offset` | `timestamp` | `timestamp-type` | `headers` | `leader-epoch`) optional.
- `build(s).properties.mapping`, a new closed `recordMapping` definition: `explode` (`path` and `as`, both required) and `columns[]` (`name` and `from` required; `cast` optional).
- `fluidVersion` accepts `"0.7.7"`.

**Changed:** `embeddedLogicPattern` no longer lists `sql` as unconditionally required. It requires `sql` unless the build has a `stream` and `parameters.flink` declares `management`, and `mapping` requires `stream` (`dependentRequired`). Every document that carried `sql` is unaffected.
- Added since the v0.27.0 copy:
- Rules about valid data: `exposes[].contract.invariants[]` (the new `invariant`, `invariantAggregate` and `invariantDependency` definitions, with `fluidSqlText` and `violationBound`), and on a column `identifierFormat`, `placeholders` and `sharedAttribute`.
- `exposes[].semantics.relationships[]` (a new `semanticRelationship` definition, with `sharedColumns`) and `dimensions[].typeParams.validityParams`; the entity type `natural` gains a description.
- `$defs/schemaRegistry` (`kind` `glue` | `confluent`), used at `stream.registry` and `binding.properties.registry`; `stream.trigger`, `stream.watermark`, `stream.checkpoint` and `stream.startingOffsets`.
- `mapping.columns[].maskAtIngest` (`hash`), and `embeddedLogicPattern.properties.parameters.spark`, a new `sparkParameters` definition (`runtime` required).
- `column.protobuf.field`, for a topic whose value is Protobuf.
- `build.engine` value `lakeflow`.
- `packaging.containers` keys `lfTagVocabulary`, `emrApplication`, `schemaRegistry`, `athenaWorkgroup`, `kmsKey` and `mwaaEnvironment`.
- `connection.password` and `auth.token` as `writeOnly` members, on an acquisition source and on an acquisition build's `deployment.auth`.

**Changed:**

- `embeddedLogicPattern` no longer lists `sql` as unconditionally required. It requires `sql` unless the build has a `stream` and either `parameters.flink` declares `management`, or it has a `mapping` and `parameters.spark`; `mapping` requires `stream` (`dependentRequired`). Every document that carried `sql` is unaffected.
- `secretRef` (an acquisition source's `connection`, the `auth` of an acquisition build's `deployment`, and the `kafka-connect` `schema_registry.auth`) is now the shared `$defs/secretRef`: one pattern per scheme (`env`, `file`, `vault`, `aws-sm`, `gcp-sm`, `azure-kv`, and other lower-case schemes). It was any string of the form `<scheme>://<anything>`. This is the narrowing: a `vault://` reference now needs a `#field`, so `vault://secret/data/crm-db` is refused and `vault://secret/data/crm-db#password` is valid. `scripts/check-compat.py` reports it and `scripts/compat-waivers.txt` records it. See the [0.7.7 notes](/fluid/releases/0.7.7#secret-references).

## 0.7.5 to 0.7.6: Declarative Packaging Modes

Expand Down
Loading
Loading