From ab0ba0827e34bd58397d674f60ef99fb09ffaa02 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Wed, 23 Sep 2026 11:45:45 -0700 Subject: [PATCH 01/10] [core-spec] Add layered query interface overview Introduces layered_query_interface.md, a short overview of the four-layer query model (Expression Language, Relational Interface, Wide Table, Ontology) for readers navigating the individual layer specs. Clarifies that Layer 1 is an interchange / model-definition language, that query languages are a per-layer concern, and that Layer 2 specifically extends Layer 1 expressions with relational SQL constructs plus MEASURE(). References both the original layered-interface proposal (Justin Talbot) and the four-layer refinement (Will Pugh), and credits both as contributors. Co-authored-by: Isaac --- core-spec/layered_query_interface.md | 126 +++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 core-spec/layered_query_interface.md diff --git a/core-spec/layered_query_interface.md b/core-spec/layered_query_interface.md new file mode 100644 index 000000000..382970d46 --- /dev/null +++ b/core-spec/layered_query_interface.md @@ -0,0 +1,126 @@ + + +# Layered Query Interface for Ossie Models + +**Status:** Draft for the Ossie Semantics Working Group. +**Contributors:** Justin Talbot, Will Pugh, Chris Eubank. + +--- + +## Overview + +Ossie consumers (humans, BI tools, and agents) need a query *interface*: a way to evaluate queries +against an Ossie model with correctness guarantees. + +A single interface cannot serve all integration styles equally well. An AI agent may want the +flexibility to generate low-level SQL queries directly from an Ossie model. A SQL-native BI tool +may want full control over query semantics while querying a shared model in a database. A data +analyst may prefer a simple declarative interface with no SQL at all. A higher-order ontology or +knowledge-graph layer may require concepts that SQL does not naturally express. + +Ossie therefore adopts a **layered query interface**—four layers that each address a distinct +correctness or expressiveness concern. Consumers choose the layer that best matches their +integration use case. Higher layers inherit correctness from lower ones, and each layer can be +adopted independently. + +This document describes the four layers at a high level. Each layer is specified in full detail in +its own document (linked below). + +--- + +## The Four Layers + +### Layer 1 — Common Expression Language + +Layer 1 defines the shared expression language used inside Ossie model definitions. Measure +definitions, calculated fields, and filter predicates in an Ossie model file are written in this +portable SQL expression subset. Layer 1 establishes the guarantee: **expressions in an Ossie model +evaluate identically on any conforming engine.** A model authored once can be deployed anywhere +without rewriting its expressions. + +**Specification:** [`expression_language.md`](expression_language.md) + +### Layer 2 — Relational Interface (SQL Measures) + +Layer 2 is the relational query interface for Ossie models. Its query language is the Layer 1 +expression language extended with the relational constructs of SQL: joins, group-by, filters, +CTEs, ordering, etc.—everything outside of scalar and aggregate expressions. To +this it adds one extension, `MEASURE()`, which evaluates a named measure at its declared grain. + +Layer 2 establishes the guarantee: **measures are evaluated without duplication regardless of +how they are queried.** Fan-out and chasm-trap errors cannot occur when measures are queried +through this interface. The model's datasets are exposed as SQL relations; consumers query them +with ordinary SQL, using `MEASURE()` in place of raw aggregates. + +This layer preserves full SQL invariants. Tools that generate SQL, embed SQL in notebooks, or +build on the relational algebra can use Layer 2 without giving up any SQL guarantees. + +**Specification:** [`relational_semantics.md`](relational_semantics.md) + +### Layer 3 — Wide Table Interface (Dimensional) + +Layer 3 exposes a multi-table Ossie model as a single flat object. The consumer specifies +dimensions, measures, and filters declaratively; the layer resolves which source tables to join, +in what order, and with what join type. No SQL generation is required from the consumer. + +This interface trades explicit query control for simplicity, making it accessible to ad-hoc users, +AI agents, and BI tools without their own data model. Measure correctness is inherited from +Layer 2: a wide-table query is rewritten into a SQL measures query (with heuristic join and filter +choices), so the same grain-safe evaluation guarantee holds. + +**Specification:** [apache/ossie#246](https://github.com/apache/ossie/pull/246) *(forthcoming)* + +### Layer 4 — Ontology + +Layer 4 sits above the other layers and covers higher-order semantic concepts—entities, +relationships, and knowledge-graph constructs that SQL does not naturally represent. It is defined +by the Ossie Ontology working group (`#ossie-ontology-wg`). + +**Specification:** [`../ontology/ontology.md`](../ontology/ontology.md) + +--- + +## Composition + +The four layers compose. Layer 3 can be implemented on top of Layer 2: a wide-table query is +rewritten into a SQL measures query. Layer 2 queries can be rewritten into Layer 1 queries by +correcting for duplication. Layer 1 queries are highly portable across engines. A deployment that +supports Layers 1–3 can expose both a SQL interface for SQL-native tools and a declarative +wide-table interface for tools or users who prefer it—with the same expression semantics and +correctness guarantees underlying both. + +Providers may support one layer or several, and at varying compliance levels. The layered model +makes partial adoption coherent: a tool that only needs SQL measures picks up Layer 2 without +taking a dependency on the wide-table semantics of Layer 3. + +--- + +## Background and Prior Work + +The layered approach was proposed by Justin Talbot in +[*OSI Layered Query Interface*](https://docs.google.com/document/d/1FOtRNBu6UqA2yeOwUa4jt1r2v45BWQJqS5_BTk34a1c/edit) +(May 2026) and refined by Will Pugh in +[*Proposed Semantics Priorities*](https://docs.google.com/document/d/1EgG5vOlJSubm01S1AGMJ4MKcMcjxBhJHenGKWMr849E/edit) +(August 2026). Will's document introduced the four-layer framing (adding the Ontology layer), +clarified the relationship between the wide-table and SQL-compliant interfaces, and proposed a +prioritization for defining semantics across layers. + +The grain-safe measure model underlying Layer 2 is grounded in Hyde and Fremlin, +[*Measures in SQL*](https://doi.org/10.48550/arXiv.2406.00251) (arXiv:2406.00251). From e51b8091e74b7354628f357280e69cd4c29130f2 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Wed, 23 Sep 2026 11:51:06 -0700 Subject: [PATCH 02/10] [core-spec] Note open PR for Layer 2 relational spec Co-authored-by: Isaac --- core-spec/layered_query_interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core-spec/layered_query_interface.md b/core-spec/layered_query_interface.md index 382970d46..36f85ffca 100644 --- a/core-spec/layered_query_interface.md +++ b/core-spec/layered_query_interface.md @@ -72,7 +72,7 @@ with ordinary SQL, using `MEASURE()` in place of raw aggregates. This layer preserves full SQL invariants. Tools that generate SQL, embed SQL in notebooks, or build on the relational algebra can use Layer 2 without giving up any SQL guarantees. -**Specification:** [`relational_semantics.md`](relational_semantics.md) +**Specification:** [`relational_semantics.md`](relational_semantics.md) *(open PR: [apache/ossie#354](https://github.com/apache/ossie/pull/354))* ### Layer 3 — Wide Table Interface (Dimensional) From 789d550e0f5499ba300e8101c186dea5071b293f Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Wed, 23 Sep 2026 12:06:45 -0700 Subject: [PATCH 03/10] [core-spec] Consistent open-PR format for Layer 3 spec reference Co-authored-by: Isaac --- core-spec/layered_query_interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core-spec/layered_query_interface.md b/core-spec/layered_query_interface.md index 36f85ffca..7b917839e 100644 --- a/core-spec/layered_query_interface.md +++ b/core-spec/layered_query_interface.md @@ -85,7 +85,7 @@ AI agents, and BI tools without their own data model. Measure correctness is inh Layer 2: a wide-table query is rewritten into a SQL measures query (with heuristic join and filter choices), so the same grain-safe evaluation guarantee holds. -**Specification:** [apache/ossie#246](https://github.com/apache/ossie/pull/246) *(forthcoming)* +**Specification:** *(open PR: [apache/ossie#246](https://github.com/apache/ossie/pull/246))* ### Layer 4 — Ontology From 071abfa7863962161791f23e532dd38f44c6d202 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Wed, 23 Sep 2026 12:07:24 -0700 Subject: [PATCH 04/10] [core-spec] Remove direct file link for Layer 2 spec, keep PR reference only Co-authored-by: Isaac --- core-spec/layered_query_interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core-spec/layered_query_interface.md b/core-spec/layered_query_interface.md index 7b917839e..4f5f1f535 100644 --- a/core-spec/layered_query_interface.md +++ b/core-spec/layered_query_interface.md @@ -72,7 +72,7 @@ with ordinary SQL, using `MEASURE()` in place of raw aggregates. This layer preserves full SQL invariants. Tools that generate SQL, embed SQL in notebooks, or build on the relational algebra can use Layer 2 without giving up any SQL guarantees. -**Specification:** [`relational_semantics.md`](relational_semantics.md) *(open PR: [apache/ossie#354](https://github.com/apache/ossie/pull/354))* +**Specification:** *(open PR: [apache/ossie#354](https://github.com/apache/ossie/pull/354))* ### Layer 3 — Wide Table Interface (Dimensional) From b4ef886ea74bfd7e4ceae7e51a4869fa34e63335 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Wed, 23 Sep 2026 14:56:39 -0700 Subject: [PATCH 05/10] [core-spec] Add mailing list discussion references to layered interface overview Expands the Background section with links to the five dev@ossie.apache.org threads that trace how the layered model emerged and converged, from Will Pugh's original foundational semantics proposal through the August debate over separating normative from opinionated behaviors, to the September convergence on the hybrid layered model. Co-authored-by: Isaac --- core-spec/layered_query_interface.md | 30 ++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/core-spec/layered_query_interface.md b/core-spec/layered_query_interface.md index 4f5f1f535..9867d3ea9 100644 --- a/core-spec/layered_query_interface.md +++ b/core-spec/layered_query_interface.md @@ -124,3 +124,33 @@ prioritization for defining semantics across layers. The grain-safe measure model underlying Layer 2 is grounded in Hyde and Fremlin, [*Measures in SQL*](https://doi.org/10.48550/arXiv.2406.00251) (arXiv:2406.00251). + +### Mailing List Discussion + +The following `dev@ossie.apache.org` threads trace how the layered model emerged and converged: + +- **[Ossie Foundational Semantics](https://lists.apache.org/thread/qo2fx0hl0cvlzn9q1ob9z7v7g81syj88)** + (Will Pugh, July 2026). Will proposed a foundational semantics spec (PR #246) covering the + wide-table query interface and asked the community for feedback and a path to a reference + implementation. + +- **[[DISCUSS] PRs #246 (Foundational Semantics) and #237 (Compliance Suite)](https://lists.apache.org/thread/mqlbbb4ndhb0qo9sf2yn60fb9tlzv59t)** + (Justin Talbot and Chris Eubank, August 2026). Raised concerns that PR #246 bundled normative + correctness guarantees (grain-safe aggregation) with opinionated choices (join paths, filter + propagation, join types) in a way that made adoption all-or-nothing for BI vendors. Argued for + separating the two into distinct layers. + +- **[[DISCUSS] SQL with measures as the Ossie BI / semantic layer interface](https://lists.apache.org/thread/0td2v8n2llqxyqw0r6w51fkt8klp8x5w)** + (Chris Eubank, August 2026) and + **[[DISCUSS] Ossie BI / semantic layer interface based on SQL with measures](https://lists.apache.org/thread/q95or2395khvs21nzmwkmwy1vpdgjy87)** + (Justin Talbot, August 2026). Proposed SQL with measures (the *Measures in SQL* model) as a + lower, SQL-native layer beneath the wide-table interface, giving SQL-fluent BI tools a path to + adopt Ossie without taking on multi-table join semantics. + +- **[[DISCUSS] Relational Query Interface spec (core-spec)](https://lists.apache.org/thread/b3nty67729wm5vc3ylnp8phgtxl90n4t)** + (Justin Talbot, September 2026). Announced PR #354, the Layer 2 specification. + +- **[[DISCUSS] Landing Ossie semantics and compliance suites](https://lists.apache.org/thread/1jhvnw8vg5fgtct347t62ycc2zkrox6l)** + (Chris Eubank, September 2026). Summarized the convergence: working-group discussion settled on + the hybrid layered model, with Layer 3 (PR #246) and Layer 2 (PR #354) as separate specs both + building on Layer 1, and Layer 3's output expressible in terms of Layer 2 primitives. From 02db1b82c3020b4b3232b8fb56da3ce7ac26ebcc Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Mon, 28 Sep 2026 10:44:53 -0700 Subject: [PATCH 06/10] [core-spec] Reframe as spec layers; rename Layer 2/3 to Relational/Wide Table Query Interface - Retitle document to "Ossie Specification Layers" - Add summary table showing the role of each layer (model definition, query interface x2, ontology workstream) - Rename Layer 2 to "Relational Query Interface" and Layer 3 to "Wide Table Query Interface" to make clear both are query interfaces - Clarify Layer 1 is a model definition concern, not a query concern - Tighten Layer 4 description as a separate workstream Co-authored-by: Isaac --- core-spec/layered_query_interface.md | 93 +++++++++++++++------------- 1 file changed, 50 insertions(+), 43 deletions(-) diff --git a/core-spec/layered_query_interface.md b/core-spec/layered_query_interface.md index 9867d3ea9..52523a34a 100644 --- a/core-spec/layered_query_interface.md +++ b/core-spec/layered_query_interface.md @@ -17,7 +17,7 @@ under the License. --> -# Layered Query Interface for Ossie Models +# Ossie Specification Layers **Status:** Draft for the Ossie Semantics Working Group. **Contributors:** Justin Talbot, Will Pugh, Chris Eubank. @@ -26,43 +26,50 @@ ## Overview -Ossie consumers (humans, BI tools, and agents) need a query *interface*: a way to evaluate queries -against an Ossie model with correctness guarantees. +Ossie is organized into four specification layers, each addressing a distinct concern: -A single interface cannot serve all integration styles equally well. An AI agent may want the -flexibility to generate low-level SQL queries directly from an Ossie model. A SQL-native BI tool -may want full control over query semantics while querying a shared model in a database. A data -analyst may prefer a simple declarative interface with no SQL at all. A higher-order ontology or -knowledge-graph layer may require concepts that SQL does not naturally express. +| Layer | Name | Role | +|-------|------|------| +| 1 | Expression Language | Model definition: the expression language used inside Ossie model definitions | +| 2 | Relational Query Interface | Query interface: SQL-native, grain-safe measure evaluation, full SQL invariants | +| 3 | Wide Table Query Interface | Query interface: declarative flat-object queries, heuristic join resolution | +| 4 | Ontology | Ontology workstream: higher-order semantic and knowledge-graph concepts | -Ossie therefore adopts a **layered query interface**—four layers that each address a distinct -correctness or expressiveness concern. Consumers choose the layer that best matches their -integration use case. Higher layers inherit correctness from lower ones, and each layer can be -adopted independently. +Layer 1 defines what an Ossie model *is*—the model format and the expression language authors +use to write it. Layers 2 and 3 are query interfaces that sit on top of a deployed model; +they differ in how much SQL knowledge they require from the consumer. Layer 4 is a separate +workstream covering ontological concepts that SQL does not naturally express. -This document describes the four layers at a high level. Each layer is specified in full detail in -its own document (linked below). +Each layer can be adopted independently. Layers 2 and 3 compose: a wide-table query (Layer 3) +can be rewritten as a SQL measures query (Layer 2), so correctness guarantees flow through. + +This document describes each layer at a high level. Each is specified in full in its own +document (linked below). --- -## The Four Layers +## The Layers -### Layer 1 — Common Expression Language +### Layer 1 — Expression Language Layer 1 defines the shared expression language used inside Ossie model definitions. Measure definitions, calculated fields, and filter predicates in an Ossie model file are written in this -portable SQL expression subset. Layer 1 establishes the guarantee: **expressions in an Ossie model -evaluate identically on any conforming engine.** A model authored once can be deployed anywhere -without rewriting its expressions. +portable SQL expression subset. Layer 1 establishes the guarantee: **expressions in an Ossie +model evaluate identically on any conforming engine.** A model authored once can be deployed +anywhere without rewriting its expressions. + +Layer 1 is a model *definition* concern, not a query concern. It does not prescribe the query +language a consumer uses when querying a deployed model—that is a per-layer decision for +Layers 2 and 3. **Specification:** [`expression_language.md`](expression_language.md) -### Layer 2 — Relational Interface (SQL Measures) +### Layer 2 — Relational Query Interface -Layer 2 is the relational query interface for Ossie models. Its query language is the Layer 1 +Layer 2 is the SQL-native query interface for Ossie models. Its query language is the Layer 1 expression language extended with the relational constructs of SQL: joins, group-by, filters, -CTEs, ordering, etc.—everything outside of scalar and aggregate expressions. To -this it adds one extension, `MEASURE()`, which evaluates a named measure at its declared grain. +CTEs, ordering, etc. To this it adds one extension, `MEASURE()`, which evaluates a named +measure at its declared grain. Layer 2 establishes the guarantee: **measures are evaluated without duplication regardless of how they are queried.** Fan-out and chasm-trap errors cannot occur when measures are queried @@ -74,24 +81,26 @@ build on the relational algebra can use Layer 2 without giving up any SQL guaran **Specification:** *(open PR: [apache/ossie#354](https://github.com/apache/ossie/pull/354))* -### Layer 3 — Wide Table Interface (Dimensional) +### Layer 3 — Wide Table Query Interface -Layer 3 exposes a multi-table Ossie model as a single flat object. The consumer specifies -dimensions, measures, and filters declaratively; the layer resolves which source tables to join, -in what order, and with what join type. No SQL generation is required from the consumer. +Layer 3 is the declarative query interface for Ossie models. It exposes a multi-table model as +a single flat object. The consumer specifies dimensions, measures, and filters; the layer +resolves which source tables to join, in what order, and with what join type. No SQL generation +is required from the consumer. -This interface trades explicit query control for simplicity, making it accessible to ad-hoc users, -AI agents, and BI tools without their own data model. Measure correctness is inherited from -Layer 2: a wide-table query is rewritten into a SQL measures query (with heuristic join and filter -choices), so the same grain-safe evaluation guarantee holds. +This interface trades explicit query control for simplicity, making it accessible to ad-hoc +users, AI agents, and BI tools without their own data model. Measure correctness is inherited +from Layer 2: a wide-table query is rewritten into a SQL measures query (with heuristic join +and filter choices), so the same grain-safe evaluation guarantee holds. **Specification:** *(open PR: [apache/ossie#246](https://github.com/apache/ossie/pull/246))* ### Layer 4 — Ontology -Layer 4 sits above the other layers and covers higher-order semantic concepts—entities, -relationships, and knowledge-graph constructs that SQL does not naturally represent. It is defined -by the Ossie Ontology working group (`#ossie-ontology-wg`). +Layer 4 covers higher-order semantic concepts—entities, relationships, and knowledge-graph +constructs that SQL does not naturally represent. It is defined by the Ossie Ontology working +group (`#ossie-ontology-wg`) and is a separate workstream from the query interfaces in Layers +2 and 3. **Specification:** [`../ontology/ontology.md`](../ontology/ontology.md) @@ -99,16 +108,14 @@ by the Ossie Ontology working group (`#ossie-ontology-wg`). ## Composition -The four layers compose. Layer 3 can be implemented on top of Layer 2: a wide-table query is -rewritten into a SQL measures query. Layer 2 queries can be rewritten into Layer 1 queries by -correcting for duplication. Layer 1 queries are highly portable across engines. A deployment that -supports Layers 1–3 can expose both a SQL interface for SQL-native tools and a declarative -wide-table interface for tools or users who prefer it—with the same expression semantics and -correctness guarantees underlying both. +Layers 2 and 3 compose: a wide-table (Layer 3) query is rewritten into a SQL measures +(Layer 2) query with heuristic join and filter choices. This means correctness guarantees +flow from Layer 2 through Layer 3—a consumer using the declarative interface gets the same +grain-safe measure evaluation as one writing SQL directly. -Providers may support one layer or several, and at varying compliance levels. The layered model -makes partial adoption coherent: a tool that only needs SQL measures picks up Layer 2 without -taking a dependency on the wide-table semantics of Layer 3. +Providers may support one query interface layer or both, at varying compliance levels. The +layered model makes partial adoption coherent: a tool that only needs SQL measures adopts +Layer 2 without taking on the wide-table join semantics of Layer 3. --- From fc3c1826f43c6184b9ec5b59c8ce923b2631b2c8 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Mon, 28 Sep 2026 10:48:39 -0700 Subject: [PATCH 07/10] [core-spec] Rename to spec_layers.md; expand Layer 1 to Model Definition MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rename layered_query_interface.md → spec_layers.md - Rename Layer 1 from "Expression Language" to "Model Definition" - Expand Layer 1 to cover both the model format (spec.md) and the expression language (expression_language.md), framing them together as the authoring/interchange layer Co-authored-by: Isaac --- ...ered_query_interface.md => spec_layers.md} | 26 +++++++++++-------- 1 file changed, 15 insertions(+), 11 deletions(-) rename core-spec/{layered_query_interface.md => spec_layers.md} (87%) diff --git a/core-spec/layered_query_interface.md b/core-spec/spec_layers.md similarity index 87% rename from core-spec/layered_query_interface.md rename to core-spec/spec_layers.md index 52523a34a..aa94a6d2d 100644 --- a/core-spec/layered_query_interface.md +++ b/core-spec/spec_layers.md @@ -30,7 +30,7 @@ Ossie is organized into four specification layers, each addressing a distinct co | Layer | Name | Role | |-------|------|------| -| 1 | Expression Language | Model definition: the expression language used inside Ossie model definitions | +| 1 | Model Definition | Model definition: the Ossie model format and the expression language used within it | | 2 | Relational Query Interface | Query interface: SQL-native, grain-safe measure evaluation, full SQL invariants | | 3 | Wide Table Query Interface | Query interface: declarative flat-object queries, heuristic join resolution | | 4 | Ontology | Ontology workstream: higher-order semantic and knowledge-graph concepts | @@ -50,19 +50,23 @@ document (linked below). ## The Layers -### Layer 1 — Expression Language +### Layer 1 — Model Definition -Layer 1 defines the shared expression language used inside Ossie model definitions. Measure -definitions, calculated fields, and filter predicates in an Ossie model file are written in this -portable SQL expression subset. Layer 1 establishes the guarantee: **expressions in an Ossie -model evaluate identically on any conforming engine.** A model authored once can be deployed -anywhere without rewriting its expressions. +Layer 1 defines what an Ossie model *is*. It covers two related specifications: -Layer 1 is a model *definition* concern, not a query concern. It does not prescribe the query -language a consumer uses when querying a deployed model—that is a per-layer decision for -Layers 2 and 3. +**Model format** ([`spec.md`](spec.md)): the structure of an Ossie model file—datasets, +relationships, fields, and metrics. This is the interchange format: the common schema that +allows semantic model definitions to be moved between tools. -**Specification:** [`expression_language.md`](expression_language.md) +**Expression language** ([`expression_language.md`](expression_language.md)): the portable SQL +expression subset used to write measure definitions, calculated fields, and filter predicates +within a model file. Layer 1 establishes the guarantee: **expressions in an Ossie model +evaluate identically on any conforming engine.** A model authored once can be deployed anywhere +without rewriting its expressions. + +Together these define the authoring layer of Ossie. Layer 1 is a model *definition* concern, +not a query concern. It does not prescribe the query language a consumer uses against a deployed +model—that is a per-layer decision for Layers 2 and 3. ### Layer 2 — Relational Query Interface From cf79792f48b64dae248b435ba9fec5caa6f4b8d1 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Mon, 5 Oct 2026 11:47:48 -0700 Subject: [PATCH 08/10] [core-spec] Mark spec layers doc as informational; define Layer 3 by Layer 2 equivalence Address review feedback: update the status line, state up front that this is an informational overview pointing to the normative layer specs, and clarify that Layer 2 is the semantic target for Layer 3 rather than a required execution path. Co-authored-by: Isaac --- core-spec/spec_layers.md | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/core-spec/spec_layers.md b/core-spec/spec_layers.md index aa94a6d2d..0ee88c8cc 100644 --- a/core-spec/spec_layers.md +++ b/core-spec/spec_layers.md @@ -19,13 +19,17 @@ # Ossie Specification Layers -**Status:** Draft for the Ossie Semantics Working Group. +**Status:** Informational / Architecture Overview (Draft). **Contributors:** Justin Talbot, Will Pugh, Chris Eubank. --- ## Overview +This document is an informational architecture overview. It describes how the Ossie +specifications fit together and links to the specification for each layer. It does not +define normative requirements itself; those are in each layer's specification. + Ossie is organized into four specification layers, each addressing a distinct concern: | Layer | Name | Role | @@ -40,11 +44,9 @@ use to write it. Layers 2 and 3 are query interfaces that sit on top of a deploy they differ in how much SQL knowledge they require from the consumer. Layer 4 is a separate workstream covering ontological concepts that SQL does not naturally express. -Each layer can be adopted independently. Layers 2 and 3 compose: a wide-table query (Layer 3) -can be rewritten as a SQL measures query (Layer 2), so correctness guarantees flow through. - -This document describes each layer at a high level. Each is specified in full in its own -document (linked below). +Each layer can be adopted independently. Layers 2 and 3 compose: the meaning of a wide-table +query (Layer 3) is defined by an equivalent SQL measures query (Layer 2), so correctness +guarantees flow through. --- @@ -94,8 +96,8 @@ is required from the consumer. This interface trades explicit query control for simplicity, making it accessible to ad-hoc users, AI agents, and BI tools without their own data model. Measure correctness is inherited -from Layer 2: a wide-table query is rewritten into a SQL measures query (with heuristic join -and filter choices), so the same grain-safe evaluation guarantee holds. +from Layer 2: a wide-table query is defined as equivalent to a SQL measures query (with +heuristic join and filter choices), so the same grain-safe evaluation guarantee holds. **Specification:** *(open PR: [apache/ossie#246](https://github.com/apache/ossie/pull/246))* @@ -112,10 +114,14 @@ group (`#ossie-ontology-wg`) and is a separate workstream from the query interfa ## Composition -Layers 2 and 3 compose: a wide-table (Layer 3) query is rewritten into a SQL measures -(Layer 2) query with heuristic join and filter choices. This means correctness guarantees -flow from Layer 2 through Layer 3—a consumer using the declarative interface gets the same -grain-safe measure evaluation as one writing SQL directly. +Layers 2 and 3 compose: the meaning of a wide-table (Layer 3) query is defined by an +equivalent SQL measures (Layer 2) query, with heuristic join and filter choices. This means +correctness guarantees flow from Layer 2 through Layer 3—a consumer using the declarative +interface gets the same grain-safe measure evaluation as one writing SQL directly. + +Layer 2 is the semantic target for Layer 3, not a required execution path. An implementation +does not have to translate Layer 3 queries into Layer 2 SQL text; it only has to return the +same results as the equivalent Layer 2 query. Providers may support one query interface layer or both, at varying compliance levels. The layered model makes partial adoption coherent: a tool that only needs SQL measures adopts From a9a9219698eb443e2cdd1c87bfea012793a5c555 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Mon, 12 Oct 2026 09:52:25 -0700 Subject: [PATCH 09/10] [core-spec] Describe Layer 2 measure extensions, multi-dataset evaluation, and Layer 3 drop-down Address remaining discussion feedback on the spec layers overview: - Layer 2 adds a measure type and in-query measure definitions, not only MEASURE(). - Clarify that Layer 1 expressions have SQL meaning over a single relation, and that the query interfaces define evaluation across a model's multiple datasets. - State that a Layer 3 consumer can write a query in Layer 2 instead. Co-authored-by: Isaac --- core-spec/spec_layers.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/core-spec/spec_layers.md b/core-spec/spec_layers.md index 0ee88c8cc..297bf10f7 100644 --- a/core-spec/spec_layers.md +++ b/core-spec/spec_layers.md @@ -70,11 +70,16 @@ Together these define the authoring layer of Ossie. Layer 1 is a model *definiti not a query concern. It does not prescribe the query language a consumer uses against a deployed model—that is a per-layer decision for Layers 2 and 3. +Over a single relation, Layer 1 expressions mean what the equivalent SQL expressions mean. +Layer 1 does not define how expressions are evaluated across a model with multiple datasets; +Layers 2 and 3 define that. + ### Layer 2 — Relational Query Interface Layer 2 is the SQL-native query interface for Ossie models. Its query language is the Layer 1 expression language extended with the relational constructs of SQL: joins, group-by, filters, -CTEs, ordering, etc. To this it adds one extension, `MEASURE()`, which evaluates a named +CTEs, ordering, etc. To this it adds a small set of extensions for measures: a measure column +type, a way to define measures within a query, and the `MEASURE()` function, which evaluates a measure at its declared grain. Layer 2 establishes the guarantee: **measures are evaluated without duplication regardless of @@ -98,6 +103,8 @@ This interface trades explicit query control for simplicity, making it accessibl users, AI agents, and BI tools without their own data model. Measure correctness is inherited from Layer 2: a wide-table query is defined as equivalent to a SQL measures query (with heuristic join and filter choices), so the same grain-safe evaluation guarantee holds. +Layers 2 and 3 query the same model, so a question that Layer 3 cannot express can be answered +in Layer 2 without changing the model. **Specification:** *(open PR: [apache/ossie#246](https://github.com/apache/ossie/pull/246))* From ada12071b71965d526e4afba827ad250dfd7beca Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Tue, 13 Oct 2026 11:48:06 -0700 Subject: [PATCH 10/10] [core-spec] Align spec layers overview with Layer 2 terminology Follow the updates to the Relational Query Interface spec (#354): the evaluation function is AGG(), and "grain-safe" becomes "duplication-free". Co-authored-by: Isaac --- core-spec/spec_layers.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/core-spec/spec_layers.md b/core-spec/spec_layers.md index 297bf10f7..b3ed7dedd 100644 --- a/core-spec/spec_layers.md +++ b/core-spec/spec_layers.md @@ -35,7 +35,7 @@ Ossie is organized into four specification layers, each addressing a distinct co | Layer | Name | Role | |-------|------|------| | 1 | Model Definition | Model definition: the Ossie model format and the expression language used within it | -| 2 | Relational Query Interface | Query interface: SQL-native, grain-safe measure evaluation, full SQL invariants | +| 2 | Relational Query Interface | Query interface: SQL-native, duplication-free measure evaluation, full SQL invariants | | 3 | Wide Table Query Interface | Query interface: declarative flat-object queries, heuristic join resolution | | 4 | Ontology | Ontology workstream: higher-order semantic and knowledge-graph concepts | @@ -79,13 +79,13 @@ Layers 2 and 3 define that. Layer 2 is the SQL-native query interface for Ossie models. Its query language is the Layer 1 expression language extended with the relational constructs of SQL: joins, group-by, filters, CTEs, ordering, etc. To this it adds a small set of extensions for measures: a measure column -type, a way to define measures within a query, and the `MEASURE()` function, which evaluates a -measure at its declared grain. +type, a way to define measures within a query, and the `AGG()` function, which evaluates a +measure. Layer 2 establishes the guarantee: **measures are evaluated without duplication regardless of how they are queried.** Fan-out and chasm-trap errors cannot occur when measures are queried through this interface. The model's datasets are exposed as SQL relations; consumers query them -with ordinary SQL, using `MEASURE()` in place of raw aggregates. +with ordinary SQL, using `AGG()` in place of raw aggregates. This layer preserves full SQL invariants. Tools that generate SQL, embed SQL in notebooks, or build on the relational algebra can use Layer 2 without giving up any SQL guarantees. @@ -102,7 +102,7 @@ is required from the consumer. This interface trades explicit query control for simplicity, making it accessible to ad-hoc users, AI agents, and BI tools without their own data model. Measure correctness is inherited from Layer 2: a wide-table query is defined as equivalent to a SQL measures query (with -heuristic join and filter choices), so the same grain-safe evaluation guarantee holds. +heuristic join and filter choices), so the same duplication-free evaluation guarantee holds. Layers 2 and 3 query the same model, so a question that Layer 3 cannot express can be answered in Layer 2 without changing the model. @@ -124,7 +124,7 @@ group (`#ossie-ontology-wg`) and is a separate workstream from the query interfa Layers 2 and 3 compose: the meaning of a wide-table (Layer 3) query is defined by an equivalent SQL measures (Layer 2) query, with heuristic join and filter choices. This means correctness guarantees flow from Layer 2 through Layer 3—a consumer using the declarative -interface gets the same grain-safe measure evaluation as one writing SQL directly. +interface gets the same duplication-free measure evaluation as one writing SQL directly. Layer 2 is the semantic target for Layer 3, not a required execution path. An implementation does not have to translate Layer 3 queries into Layer 2 SQL text; it only has to return the @@ -146,7 +146,7 @@ The layered approach was proposed by Justin Talbot in clarified the relationship between the wide-table and SQL-compliant interfaces, and proposed a prioritization for defining semantics across layers. -The grain-safe measure model underlying Layer 2 is grounded in Hyde and Fremlin, +The measure model underlying Layer 2 is grounded in Hyde and Fremlin, [*Measures in SQL*](https://doi.org/10.48550/arXiv.2406.00251) (arXiv:2406.00251). ### Mailing List Discussion @@ -160,7 +160,7 @@ The following `dev@ossie.apache.org` threads trace how the layered model emerged - **[[DISCUSS] PRs #246 (Foundational Semantics) and #237 (Compliance Suite)](https://lists.apache.org/thread/mqlbbb4ndhb0qo9sf2yn60fb9tlzv59t)** (Justin Talbot and Chris Eubank, August 2026). Raised concerns that PR #246 bundled normative - correctness guarantees (grain-safe aggregation) with opinionated choices (join paths, filter + correctness guarantees (duplication-free aggregation) with opinionated choices (join paths, filter propagation, join types) in a way that made adoption all-or-nothing for BI vendors. Argued for separating the two into distinct layers.