diff --git a/core-spec/spec_layers.md b/core-spec/spec_layers.md new file mode 100644 index 000000000..b3ed7dedd --- /dev/null +++ b/core-spec/spec_layers.md @@ -0,0 +1,180 @@ + + +# Ossie Specification Layers + +**Status:** Informational / Architecture Overview (Draft). +**Contributors:** Justin Talbot, Will Pugh, Chris Eubank. + +--- + +## Overview + +This document is an informational architecture overview. It describes how the Ossie +specifications fit together and links to the specification for each layer. It does not +define normative requirements itself; those are in each layer's specification. + +Ossie is organized into four specification layers, each addressing a distinct concern: + +| Layer | Name | Role | +|-------|------|------| +| 1 | Model Definition | Model definition: the Ossie model format and the expression language used within it | +| 2 | Relational Query Interface | Query interface: SQL-native, duplication-free measure evaluation, full SQL invariants | +| 3 | Wide Table Query Interface | Query interface: declarative flat-object queries, heuristic join resolution | +| 4 | Ontology | Ontology workstream: higher-order semantic and knowledge-graph concepts | + +Layer 1 defines what an Ossie model *is*—the model format and the expression language authors +use to write it. Layers 2 and 3 are query interfaces that sit on top of a deployed model; +they differ in how much SQL knowledge they require from the consumer. Layer 4 is a separate +workstream covering ontological concepts that SQL does not naturally express. + +Each layer can be adopted independently. Layers 2 and 3 compose: the meaning of a wide-table +query (Layer 3) is defined by an equivalent SQL measures query (Layer 2), so correctness +guarantees flow through. + +--- + +## The Layers + +### Layer 1 — Model Definition + +Layer 1 defines what an Ossie model *is*. It covers two related specifications: + +**Model format** ([`spec.md`](spec.md)): the structure of an Ossie model file—datasets, +relationships, fields, and metrics. This is the interchange format: the common schema that +allows semantic model definitions to be moved between tools. + +**Expression language** ([`expression_language.md`](expression_language.md)): the portable SQL +expression subset used to write measure definitions, calculated fields, and filter predicates +within a model file. Layer 1 establishes the guarantee: **expressions in an Ossie model +evaluate identically on any conforming engine.** A model authored once can be deployed anywhere +without rewriting its expressions. + +Together these define the authoring layer of Ossie. Layer 1 is a model *definition* concern, +not a query concern. It does not prescribe the query language a consumer uses against a deployed +model—that is a per-layer decision for Layers 2 and 3. + +Over a single relation, Layer 1 expressions mean what the equivalent SQL expressions mean. +Layer 1 does not define how expressions are evaluated across a model with multiple datasets; +Layers 2 and 3 define that. + +### Layer 2 — Relational Query Interface + +Layer 2 is the SQL-native query interface for Ossie models. Its query language is the Layer 1 +expression language extended with the relational constructs of SQL: joins, group-by, filters, +CTEs, ordering, etc. To this it adds a small set of extensions for measures: a measure column +type, a way to define measures within a query, and the `AGG()` function, which evaluates a +measure. + +Layer 2 establishes the guarantee: **measures are evaluated without duplication regardless of +how they are queried.** Fan-out and chasm-trap errors cannot occur when measures are queried +through this interface. The model's datasets are exposed as SQL relations; consumers query them +with ordinary SQL, using `AGG()` in place of raw aggregates. + +This layer preserves full SQL invariants. Tools that generate SQL, embed SQL in notebooks, or +build on the relational algebra can use Layer 2 without giving up any SQL guarantees. + +**Specification:** *(open PR: [apache/ossie#354](https://github.com/apache/ossie/pull/354))* + +### Layer 3 — Wide Table Query Interface + +Layer 3 is the declarative query interface for Ossie models. It exposes a multi-table model as +a single flat object. The consumer specifies dimensions, measures, and filters; the layer +resolves which source tables to join, in what order, and with what join type. No SQL generation +is required from the consumer. + +This interface trades explicit query control for simplicity, making it accessible to ad-hoc +users, AI agents, and BI tools without their own data model. Measure correctness is inherited +from Layer 2: a wide-table query is defined as equivalent to a SQL measures query (with +heuristic join and filter choices), so the same duplication-free evaluation guarantee holds. +Layers 2 and 3 query the same model, so a question that Layer 3 cannot express can be answered +in Layer 2 without changing the model. + +**Specification:** *(open PR: [apache/ossie#246](https://github.com/apache/ossie/pull/246))* + +### Layer 4 — Ontology + +Layer 4 covers higher-order semantic concepts—entities, relationships, and knowledge-graph +constructs that SQL does not naturally represent. It is defined by the Ossie Ontology working +group (`#ossie-ontology-wg`) and is a separate workstream from the query interfaces in Layers +2 and 3. + +**Specification:** [`../ontology/ontology.md`](../ontology/ontology.md) + +--- + +## Composition + +Layers 2 and 3 compose: the meaning of a wide-table (Layer 3) query is defined by an +equivalent SQL measures (Layer 2) query, with heuristic join and filter choices. This means +correctness guarantees flow from Layer 2 through Layer 3—a consumer using the declarative +interface gets the same duplication-free measure evaluation as one writing SQL directly. + +Layer 2 is the semantic target for Layer 3, not a required execution path. An implementation +does not have to translate Layer 3 queries into Layer 2 SQL text; it only has to return the +same results as the equivalent Layer 2 query. + +Providers may support one query interface layer or both, at varying compliance levels. The +layered model makes partial adoption coherent: a tool that only needs SQL measures adopts +Layer 2 without taking on the wide-table join semantics of Layer 3. + +--- + +## Background and Prior Work + +The layered approach was proposed by Justin Talbot in +[*OSI Layered Query Interface*](https://docs.google.com/document/d/1FOtRNBu6UqA2yeOwUa4jt1r2v45BWQJqS5_BTk34a1c/edit) +(May 2026) and refined by Will Pugh in +[*Proposed Semantics Priorities*](https://docs.google.com/document/d/1EgG5vOlJSubm01S1AGMJ4MKcMcjxBhJHenGKWMr849E/edit) +(August 2026). Will's document introduced the four-layer framing (adding the Ontology layer), +clarified the relationship between the wide-table and SQL-compliant interfaces, and proposed a +prioritization for defining semantics across layers. + +The measure model underlying Layer 2 is grounded in Hyde and Fremlin, +[*Measures in SQL*](https://doi.org/10.48550/arXiv.2406.00251) (arXiv:2406.00251). + +### Mailing List Discussion + +The following `dev@ossie.apache.org` threads trace how the layered model emerged and converged: + +- **[Ossie Foundational Semantics](https://lists.apache.org/thread/qo2fx0hl0cvlzn9q1ob9z7v7g81syj88)** + (Will Pugh, July 2026). Will proposed a foundational semantics spec (PR #246) covering the + wide-table query interface and asked the community for feedback and a path to a reference + implementation. + +- **[[DISCUSS] PRs #246 (Foundational Semantics) and #237 (Compliance Suite)](https://lists.apache.org/thread/mqlbbb4ndhb0qo9sf2yn60fb9tlzv59t)** + (Justin Talbot and Chris Eubank, August 2026). Raised concerns that PR #246 bundled normative + correctness guarantees (duplication-free aggregation) with opinionated choices (join paths, filter + propagation, join types) in a way that made adoption all-or-nothing for BI vendors. Argued for + separating the two into distinct layers. + +- **[[DISCUSS] SQL with measures as the Ossie BI / semantic layer interface](https://lists.apache.org/thread/0td2v8n2llqxyqw0r6w51fkt8klp8x5w)** + (Chris Eubank, August 2026) and + **[[DISCUSS] Ossie BI / semantic layer interface based on SQL with measures](https://lists.apache.org/thread/q95or2395khvs21nzmwkmwy1vpdgjy87)** + (Justin Talbot, August 2026). Proposed SQL with measures (the *Measures in SQL* model) as a + lower, SQL-native layer beneath the wide-table interface, giving SQL-fluent BI tools a path to + adopt Ossie without taking on multi-table join semantics. + +- **[[DISCUSS] Relational Query Interface spec (core-spec)](https://lists.apache.org/thread/b3nty67729wm5vc3ylnp8phgtxl90n4t)** + (Justin Talbot, September 2026). Announced PR #354, the Layer 2 specification. + +- **[[DISCUSS] Landing Ossie semantics and compliance suites](https://lists.apache.org/thread/1jhvnw8vg5fgtct347t62ycc2zkrox6l)** + (Chris Eubank, September 2026). Summarized the convergence: working-group discussion settled on + the hybrid layered model, with Layer 3 (PR #246) and Layer 2 (PR #354) as separate specs both + building on Layer 1, and Layer 3's output expressible in terms of Layer 2 primitives.