From 62965a65a63c5c02a6c15efc707175c6b65f53fc Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Wed, 2 Sep 2026 16:21:32 -0700 Subject: [PATCH 1/8] [core-spec] Add Relational Query Interface specification Adds core-spec/relational_semantics.md, a draft specification for querying Ossie semantic models through SQL extended with measure semantics -- Layer 2 of the layered query interface (the Relational Interface). --- core-spec/relational_semantics.md | 1254 +++++++++++++++++++++++++++++ 1 file changed, 1254 insertions(+) create mode 100644 core-spec/relational_semantics.md diff --git a/core-spec/relational_semantics.md b/core-spec/relational_semantics.md new file mode 100644 index 000000000..d04a3b1a4 --- /dev/null +++ b/core-spec/relational_semantics.md @@ -0,0 +1,1254 @@ + + +# Relational Query Interface for Ossie Models + +**Specification (draft)** + +Status: Draft for the Ossie Semantics Working Group. +Target: Apache Ossie (Open Semantic Interchange) semantic-model spec version `0.2.0.dev0`. +Scope: how a semantic layer exposes an Ossie model to SQL-native BI and AI tools as a set of +related SQL relations, and the semantics and correctness guarantees of querying the measures on +those relations through SQL. +Layer: this specification defines **Layer 2**, the Relational Interface, of the layered +Ossie query interface (Section 1.1). It is self-contained; Layers 1 and 3 are referenced but +specified separately. + +This specification is grounded in the measure model of Hyde and Fremlin, *Measures in SQL* +(arXiv:2406.00251): a measure is a column that attaches a calculation to a relation, is +evaluated in the context of the query that references it, and expands transparently into ordinary +SQL. This document adapts that model to the multi-table, cross-tool setting of Ossie. + +--- + +## 1. Introduction + +### 1.1 Purpose + +Ossie has focused on interchange: moving a semantic model's definition between tools. But +consumers also need a way to query a model and get back correct, consistent metrics, the same for +every tool that queries it. + +Different consumers need different things from a query interface, so Ossie is adopting a layered +approach. This document specifies **Layer 2, the Relational Interface**: a SQL-based query +interface for BI and AI tools that generate their own SQL and want its full power. Its +counterpart, **Layer 3, the Wide Table Interface**, flattens the model into a single object that a +consumer queries without writing joins. That is simpler but more restricted, and it is proposed +separately in [apache/ossie#246](https://github.com/apache/ossie/pull/246). **Layer 1**, the +[Common Expression Language](https://github.com/apache/ossie/blob/main/core-spec/expression_language.md), +is the portable SQL expression subset the higher layers build on. A higher ontology layer +(Layer 4) sits above these and is out of scope here. + +Most BI and AI tools already use SQL as their primary query interface. They rely on its +expressions, filtering, grouping, and joining for slice-and-dice analysis and multi-table +modeling. So the natural interface for these tools is to expose an Ossie model's datasets as +relations and let them query with ordinary SQL. + +Plain SQL, though, cannot guarantee correct measures. Aggregating across joined tables is +error-prone: fan-out and chasm traps silently double-count. And different tools each compute the +same metric their own way, so results are often neither correct nor consistent across tools. SQL +alone gives flexibility, but not correctness. + +This specification proposes that BI and AI tools query Ossie models through standard SQL extended +with correct measure semantics ([*Measures in SQL*](https://arxiv.org/abs/2406.00251)). The engine +evaluates a measure at its own grain, and it stays correct however the consumer filters, groups, +and joins. A tool evaluates it with a single `MEASURE()` aggregate and otherwise writes the SQL it +always would. Consumers get both governed, consistent measures and the flexibility of a SQL query +interface. + +### 1.2 At a glance + +A consumer connects to a semantic layer, discovers a model's relations, and queries them with +ordinary SQL. The one difference from querying plain tables: a measure is evaluated through the +`MEASURE()` aggregate: + +```sql +-- illustrative +SELECT o.region, + MEASURE(o.total_revenue), + MEASURE(s.ticket_count), + MEASURE(o.total_revenue) / MEASURE(s.ticket_count) AS revenue_per_ticket +FROM model.orders o +JOIN model.support s ON o.customer_id = s.customer_id +WHERE o.order_date >= DATE '2025-01-01' +GROUP BY o.region +ORDER BY MEASURE(o.total_revenue) DESC; +``` + +`total_revenue` (on `orders`) and `ticket_count` (on `support`) are measures at different grains, +identified as measures from metadata (Section 11). `region`, `customer_id`, and `order_date` are +ordinary fields. The tool joins the two tables as it would any tables and evaluates a measure from +each with `MEASURE()`. The engine computes each measure at its own grain, so the many-to-many +join on `customer_id` inflates neither one (no fan-out or chasm trap, G2), while `WHERE` and +`GROUP BY` slice them like any aggregate (G1), on any field, join key or not. The same query +yields the same measure values across tools. A consuming tool writes `MEASURE(name)` where it +would otherwise write a raw aggregate, and generates SQL as it always has. Its obligations are in +Section 8.5. + +For instance, the query might return: + +``` +region | total_revenue | ticket_count | revenue_per_ticket +West | 1,200,000 | 400 | 3,000 +East | 800,000 | 250 | 3,200 +``` + +Each order's revenue is counted once and each ticket once, though the join pairs every order +with several tickets and vice versa. A raw `SUM`/`COUNT` over the joined rows would inflate both. + +### 1.3 Scope + +This specification covers: + +- The **data model**: relations carrying fields and measures, and the *grain* of a + measure (Section 3). +- The **type discipline** for measures (Section 4). +- The **semantics** of defining measures and of querying them through SQL operators: + projection, selection, joins, grouping, subqueries, and set operators (Sections 5-7). +- The two **correctness guarantees** a conforming engine provides, stated normatively + (Section 8), and a non-normative reference evaluation strategy that satisfies them + (Section 9). +- The **multi-table** consistency contract and the **discovery** mechanisms by which a tool + learns a model's tables, measures, and relationships (Sections 10-11). +- **Conformance** criteria (Section 12), and out-of-scope behaviors and open questions + (Section 13). + +It does not cover: + +- Automatic join resolution: choosing which tables to join, the join types, and how filters + propagate. At Layer 2 the tool writes its own joins; resolving them automatically is a Layer 3 + concern (Section 1.1), and the choice of joins and filters is the tool's, not something + standardized here (Section 10.2). +- The model interchange format, the subject of other Ossie specifications. +- Physical execution, storage, indexing, or performance. + +The SQL spelling shown throughout is illustrative and provisional. This specification aims to +standardize the query extension: the measure column type, `MEASURE()`, and the measure-definition +marker. Its syntax is expected to be refined during working-group review rather than left for each +engine to reinvent. + +### 1.4 Requirements language + +The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, MAY, and +OPTIONAL are to be interpreted as described in RFC 2119. "The engine" denotes a conforming +implementation of this interface. "A consumer" or "a tool" denotes a client that issues +queries against it. + +--- + +## 2. Design principles + +These principles motivate the normative rules and are themselves non-normative. + +- **P1. A transparent relational interface.** An Ossie semantic model is exposed as a set of + related SQL relations. Its datasets and relationships map to relations the tool sees and queries + directly, with nothing about the multi-table structure hidden. Consumers query them with + ordinary SQL, using its standard syntax and semantics. A relation carrying measures behaves like + any other relation: it composes under joins, filters, projections, and subqueries, so standard + SQL rewrites and optimizations remain sound over it (Section 8.4). +- **P2. Correct measures.** The engine owns how a measure is evaluated. A consumer references it + by name and never reconstructs its computation, so a governed metric comes out correct and + identical across every consumer, however the consumer filters, groups, and joins tables. +- **P3. Minimal consumer change.** Querying a measure differs from an ordinary aggregate by + exactly one substitution: a raw aggregate expression becomes `MEASURE(name)`. A tool's query + generation is otherwise unchanged. The placement of aggregates in SELECT, HAVING, and ORDER BY, + and the use of subqueries, joins, and grouping, all carry over as written. + +--- + +## 3. Data model + +This interface exposes an Ossie semantic model as SQL relations. Each dataset in an Ossie model +becomes a relation, and the relation's columns are the dataset's fields and metrics. This section +defines that relational data model: relations of fields and measures (Section 3.1), the grain a +measure carries (Section 3.2), and how it maps to the Ossie model (Section 3.4). + +### 3.1 Relations, fields, measures + +A **relation** is a multiset of rows over a fixed set of named **columns**, following standard +SQL multiset semantics. Each column is exactly one of: + +- a **field**: an ordinary SQL column of some scalar type `T`; or +- a **measure**: a column of type `T measure` (Section 4) that carries an unevaluated + aggregate calculation together with the grain at which it is to be evaluated. + +A **measure relation** is a relation with at least one measure column. A relation with no +measure columns is an ordinary relation and MUST behave exactly like a standard SQL relation +under all operators; this specification imposes no additional requirements on it. + +Fields and measures share a single column namespace within a relation. Column names within +a relation MUST be unique. + +### 3.2 Grain (row identity) + +Every measure carries the identity of the source rows it aggregates. This identity is its +**grain**. At the point where a measure is defined (Section 5), each row of the measure's source +has an identity. This **row identity** is remembered as the measure propagates through a query, +and it governs how the measure is aggregated (Section 8). + +Two properties define row identity: + +- **Distinct source rows have distinct identities**, and rows that share an identity are the + same source row, identical in all content the measure reads, including the inputs its aggregate + reads. (It follows that a measure's aggregate inputs are functionally determined by its row + identity: a candidate identity under which two genuinely different rows collide is not a valid + identity.) +- A row **may lack an identity**. In a consuming query this happens to rows a join fabricates + that correspond to no source row (for example, the null-padded side of an outer join). + +A measure's row identity is fixed at definition time and is not altered by a consuming query. + +An engine may represent a row identity in different ways: an author-declared key (a dataset's +`primary_key`; Section 3.4), a synthesized id, or a physical row identifier. How it does so, and +whether any representation is exposed to consumers, are discussed in Sections 9 and 11. Whether +row identity needs a declaration syntax at all is an open question (Section 13.1). It is a +semantic notion and need not be a stored column; Section 9 gives one concrete realization. + +> **Rationale (non-normative).** The grain is the single piece of state that distinguishes a +> measure from a raw aggregate. A raw aggregate sums whatever rows reach it, so a join that +> duplicates rows silently inflates downstream aggregates. A measure remembers the identity of +> its defining rows and so can be de-duplicated back to its own grain no matter how upstream +> operations multiply rows. + +### 3.3 Multi-grain relations + +Distinct measures in the same relation MAY have distinct grains. A relation formed by joining +two measure relations (Section 7.6) carries measures at each contributing relation's original +grain. Each measure is evaluated at its own grain independently (Guarantee G2). + +### 3.4 Relationship to the Ossie semantic model + +The relations, fields, and measures above are how this interface presents an Ossie +`semantic_model` (its `datasets`, `relationships`, and `metrics`) to SQL consumers, in SQL-native +names. The correspondence: + +| This spec | Apache Ossie | Notes | +|---|---|---| +| relation | `dataset` | A logical table (fact or dimension) exposed as a SQL relation. | +| field | `field` | A scalar (non-measure) column; we adopt Ossie's term. Ossie's optional `dimension` role (e.g. `is_time`) is metadata this interface does not require. | +| measure | `metric` | An aggregate expression on the model's data (of any scalar result type). Ossie currently places metrics at the *model* level, not on a dataset; Section 10.3 reconciles that and recommends Ossie add per-relation measures. | +| grain / row identity (Section 3.2) | `primary_key` | A dataset's declared primary key is a natural source of a measure's row identity. | +| join relationship (Section 11.3) | `relationship` (`from`/`to`, `from_columns`/`to_columns`) | `from`/`to` map to source/target and the columns to the join predicate (`from` many, `to` one); Section 11.3 adds cardinality, roles, and arbitrary `ON`. | +| Layer 1 expression | `expression` (per dialect) | Ossie carries SQL per dialect; a Layer 1 expression is the portable form (Section 1.1). | + +Ossie model metadata this interface does not natively surface (`ai_context`, `custom_extensions`, +and non-selected expression dialects) is addressed under discovery (Section 11) and out of scope +(Section 13.2). A conforming engine MUST preserve metric +semantics: a metric evaluated via `MEASURE()` is computed from the model's metric definition (its +aggregate and grain), so two consumers issuing the same query agree on its value (Section 10.2). + +--- + +## 4. Type discipline + +### 4.1 The measure type + +For each scalar type `T`, there is a distinct type `T measure`. A value of type `T measure` is +an unevaluated aggregate calculation whose evaluation yields a value of type `T`. + +`T measure` is **not** a subtype of `T` and is **not** interchangeable with it. The two +central rules: + +- **R1.** `T measure` is a new type, and SQL's operators and functions are not defined on it. A + value of type `T measure` therefore MUST NOT appear where an operator or clause would have to + interpret it: as an operand of a scalar operator or function; as a `WHERE`, join, or `HAVING` + predicate (which must be `boolean`, not `boolean measure`); or as a `GROUP BY` key (grouping is + not defined over `T measure`). A measure column MAY still pass through a select list, which + accepts a value of any type, carrying its `T measure` type onward (Section 7.1). +- **R2.** The only operation that converts `T measure` to `T` is the `MEASURE` aggregate + function (Section 6), which is valid only in an aggregate context. + +A well-typed query MUST reject, at analysis time, any expression that violates R1 or R2. For +example (non-normative), if `total_sales` has type `int measure`: + +```sql +-- rejected: '+' is not defined on (int measure, int) +SELECT total_sales + 1 FROM v; + +-- rejected: grouping is not defined over int measure +SELECT total_sales FROM v GROUP BY total_sales; + +-- rejected: a join condition must be boolean; int measure is not +... JOIN w ON v.total_sales = w.total_sales; +``` + +### 4.2 Producing a measure + +A value of type `T measure` is produced only by a **measure definition** (Section 5), the +construct that marks an aggregate expression as a measure. That is also the one place an +aggregate expression may appear outside an aggregated query; everywhere else an aggregate, +including `MEASURE()`, is valid only in an aggregated context (R2). + +### 4.3 Propagation + +Under projection, selection, joins, and subqueries, a measure column retains its type +`T measure` and its grain (Sections 3.2, 7). + +--- + +## 5. Defining measures (illustrative syntax) + +### 5.1 Base measures + +A base measure attaches an aggregate expression to a relation. Illustratively, a `TO MEASURE` +marker in a projection promotes an aggregate expression to an unevaluated measure: + +```sql +-- illustrative +CREATE VIEW sales_v AS +SELECT + o_orderdate AS order_date, -- field + c_region AS region, -- field + COUNT(*) TO MEASURE AS order_count, -- measure: int measure + SUM(o_totalprice) TO MEASURE AS total_revenue -- measure: decimal measure +FROM orders JOIN customers ON o_customerkey = c_customerkey; +``` + +Normatively: + +- The operand of a measure definition MUST be a valid SQL aggregate expression (in the + Layer 1 expression language). A non-aggregate operand MUST be rejected. +- The measure's grain (Section 3.2) is captured from the `FROM` clause in effect at the point + of definition. The definition's own fields constrain nothing about the grain; the grain + is the identity of the source rows, not the projected fields. +- The result column has type `T measure`, where `T` is the aggregate expressions's output type. + +### 5.2 Derived measures + +A measure definition's expression is an ordinary SQL aggregate expression, so every value in it +comes from an aggregate function: `MEASURE()` reads an existing measure, just as `SUM` or `COUNT` +aggregates a raw field. A definition that reads one or more existing measures with `MEASURE()` +yields a derived measure: + +```sql +-- illustrative +avg_order_value := (MEASURE(total_revenue) / COUNT(DISTINCT order_id)) TO MEASURE +``` + +`MEASURE(total_revenue)` reads the existing revenue measure and `COUNT(DISTINCT order_id)` +counts orders from a raw field. A derived measure delegates to the measures it references: each +is evaluated at its own grain, and the surrounding scalar expression is applied after those +evaluations (see Section 7.2). Its inner `MEASURE()` calls are captured unevaluated, just as +`TO MEASURE` captures `SUM(x)`: they are evaluated at query time, in the consuming query's +context (not when the derived measure is defined), so no `MEASURE()` is evaluated outside an +aggregated context (R2). Derived measures MAY reference measures from more than one relation; +such a measure is a **multi-table (cross-grain) measure** (Section 10.3). + +### 5.3 Grain and window controls + +Because a measure carries its own grain, an engine MAY offer controls that manipulate that +grain: including or excluding fields from the evaluation grain, semi-additive selection, and +windowing (cumulative, trailing, period-over-period). These controls are extensions layered on +top of the base semantics defined here; their specification is out of scope for this document. +When present, they MUST preserve the guarantees of Section 8 for the grain they declare. + +--- + +## 6. Querying measures: the MEASURE function + +A measure is evaluated through the aggregate function `MEASURE`: + +- **Signature.** `MEASURE(m)` takes an argument of type `T measure` and returns a value of + type `T`. +- **Kind.** `MEASURE` is an aggregate function. It MUST be treated syntactically and + semantically as an aggregate everywhere: it MAY appear wherever a standard aggregate may + appear (e.g. SELECT list, HAVING, ORDER BY), and it MUST NOT appear where a standard aggregate + may not. Placing it in a SELECT list or `HAVING` makes the query aggregated + (Section 7.2), just as any aggregate does. +- **Argument.** The argument MUST resolve to a measure column (a column of type `T measure`), + optionally qualified by relation alias (`MEASURE(o.revenue)`). It MUST NOT be an arbitrary + scalar expression. + +`MEASURE(m)` evaluates `m`'s calculation at `m`'s grain, within the evaluation context +established by the enclosing query's filters and grouping (Sections 7-8). + +> An engine MAY expose an additional spelling of this function (for example `AGG`) as a +> synonym. Synonyms are non-normative. + +--- + +## 7. Operator semantics + +This section specifies how **measure columns** behave under each SQL operator. The semantics +follow from the definitions of a measure column and the `MEASURE()` function: + +- A measure column has the new type `T measure`, which SQL's operators and functions are not + defined on. By R1 (Section 4.1) it cannot be an operand of a scalar operator or function, nor + be used where a specific type is required, e.g. a `WHERE` or join predicate must be `boolean` (not + `boolean measure`). Grouping is not defined over `T measure`. It passes + through a select list unchanged, keeping its `T measure` type. +- `MEASURE()` is an aggregate function (Section 6). Ordinary SQL then fixes where it may appear: + in a `SELECT` list, `HAVING`, or `ORDER BY` (and inside scalar expressions there), and not in + `WHERE` or `GROUP BY`, just as `SUM(...)` may not. +- Once `MEASURE()` has evaluated a measure to a field of type `T`, the result is an ordinary + column, with no special rules thereafter. + +Every operator then behaves as in standard SQL. The only measure-specific behavior is that each +`MEASURE()` is evaluated at its measure's own grain (Section 8.1). That is what makes filtering, +grouping, and joining compose correctly (G1, G2). The subsections state only that grain behavior +and the few outright exceptions; the rest is standard SQL and is not restated. + +The following table summarizes; the subsections are normative. + +| Operator | Measure columns | +| :-- | :-- | +| SELECT, unaggregated (7.1) | Pass through as `T measure`; may be renamed; `*` includes them; may be (re)defined. Evaluating one via `MEASURE()` makes the query aggregated (7.2). | +| SELECT, aggregated (7.2) | Measures allowed only inside `MEASURE()`. A bare (unevaluated) measure in an aggregated select is an error. | +| WHERE (7.3) | Ordinary predicate; filters the measure's input rows before evaluation. `MEASURE()` only via a scalar subquery. | +| GROUP BY / GROUPING SETS / CUBE / ROLLUP (7.4) | Grouping ranges over fields; grouping is not defined over `T measure`. `GROUP BY ALL` groups by the non-measure select expressions. | +| HAVING / ORDER BY (7.5) | `MEASURE()` allowed, as for any aggregate. | +| JOIN (7.6) | Any join type/condition over fields. Each measure retains its own grain and is de-duplicated before evaluation (G2). Joining on a measure is an error. | +| Subquery / CTE (7.7) | Measures propagate as `T measure` with grain intact. | +| Set operators, DISTINCT (7.8) | Error when an input carries an unevaluated measure. | + +### 7.1 SELECT (unaggregated) + +In a SELECT without aggregation, a measure column is treated like any other column: + +- if projected (by name, renamed, or via `*`), it passes through, keeping its `T measure` + type and its grain; if not projected, it is dropped; +- it MAY be defined or derived by a measure definition in the projection (Section 5); +- it MUST NOT be the operand of a scalar operator or function (R1); +- it is not evaluated to a value here: `MEASURE()` is the only way to evaluate a measure, and a + `MEASURE()` in the select list makes the query aggregated (Section 7.2), so a genuinely + unaggregated projection contains no `MEASURE()` (R2). + +Non-normative examples: + +```sql +SELECT date_dim AS d, total_sales AS ts FROM v; -- ok: passes a measure through, renamed +SELECT * FROM v; -- ok: fields and measures pass through +SELECT total_sales + 1 FROM v; -- rejected: scalar op on a measure (R1) +``` + +### 7.2 SELECT (aggregated) + +A SELECT is aggregated (as in standard SQL) if it has a `GROUP BY` clause, `HAVING` clause, or an +aggregate expression in its select list. In an aggregated SELECT: + +- A measure column MUST appear only inside `MEASURE()`. A bare, unevaluated measure in the + select list of an aggregated query MUST be rejected. +- `MEASURE(m)` returns a scalar of type `T`; it MAY be combined with other scalars and + aggregates in the select list, HAVING, and ORDER BY. +- For a derived measure or an expression combining several `MEASURE()` calls, each + `MEASURE()` is evaluated independently at its own grain and the surrounding scalar expression + applied per output group (Section 8.1). For example, `MEASURE(revenue) / MEASURE(ticket_count)` + divides two independently grain-correct aggregates within each group. + +Non-normative examples: + +```sql +SELECT region, MEASURE(total_revenue) FROM v GROUP BY region; -- ok +SELECT MEASURE(total_sales) FROM v; -- ok: aggregated grand total +SELECT region, MAX(order_date), MEASURE(total_revenue) FROM v +GROUP BY region; -- ok: an ordinary aggregate over a field, beside a measure +SELECT region, total_revenue FROM v GROUP BY region; -- rejected: bare measure (R1) +``` + +### 7.3 WHERE + +`WHERE` filters the input rows before aggregation, so a measure sees only the surviving rows: +filtering on a field changes a measure just as it would a raw aggregate over the same rows +(half of Guarantee G1, Section 8.2). A scalar subquery in the predicate MAY use `MEASURE()`; a +predicate on a measure's value goes in `HAVING`. + +Non-normative examples: + +```sql +SELECT region, MEASURE(total_revenue) FROM v +WHERE order_date >= DATE '2025-01-01' GROUP BY region; -- ok: WHERE filters the measure's input +SELECT region, MEASURE(total_revenue) FROM v +WHERE MEASURE(total_revenue) > 100 GROUP BY region; -- rejected: aggregate in WHERE; use HAVING +SELECT region, MEASURE(total_revenue) FROM v +WHERE (SELECT MEASURE(gross) FROM budget) > 100 GROUP BY region; -- ok: MEASURE() in a subquery +``` + +### 7.4 GROUP BY + +- Grouping (`GROUP BY`, `GROUPING SETS`, `CUBE`, `ROLLUP`) ranges over fields; grouping is not + defined over `T measure` (R1). The grouping sets the grain at which each measure is reported, + the other half of Guarantee G1 (Section 8.2). +- Under `GROUP BY ALL`, measures are not grouping columns: a bare (unevaluated) measure in the + select list has nothing to group by and MUST be rejected; otherwise `ALL` groups by the + remaining fields, as usual. + +Non-normative examples: + +```sql +SELECT region, MEASURE(total_revenue) FROM v GROUP BY region; -- ok +SELECT region, MEASURE(total_revenue) FROM v GROUP BY ALL; -- ok, groups by region +SELECT region FROM v GROUP BY total_revenue; -- rejected: grouping not defined over a measure (R1) +``` + +### 7.5 HAVING and ORDER BY + +Since `MEASURE()` is an aggregate (Section 6), `HAVING` and `ORDER BY` MAY contain `MEASURE()`, +and scalar expressions over it, just as they may `SUM(...)`. + +Non-normative example: + +```sql +SELECT region, MEASURE(total_revenue) FROM v GROUP BY region +HAVING MEASURE(total_revenue) > 100 ORDER BY MEASURE(total_revenue) DESC; -- ok +``` + +### 7.6 JOIN + +A measure relation MAY be joined to any relation using any join type (INNER, LEFT, RIGHT, FULL +OUTER, CROSS) and any join condition over fields (R1). + +A join is logically applied before measures are evaluated. It affects a measure's input rows +in three ways, each handled by the evaluation semantics (Section 8.1) so the measure is still +evaluated at its own grain: + +- **Filtering.** Rows the join drops (an inner join's non-matches) are filtered from the + measure's input, just as a `WHERE` filter would, and that filtering is reflected in the measure. +- **Duplication.** A source row that matches several rows on the other side appears in several + result rows; the duplicates are collapsed before aggregation, so the measure is not inflated. +- **Null-extension.** An outer join's null-padded rows carry no row identity for the measure + (they correspond to no source row); they are dropped from this measure's aggregation input + (not from the query result; Section 8.1). + +The result of a join of measure relations is itself a measure relation whose measures retain +their original grains (Section 3.3), so joins compose. + +Non-normative example (many-to-many join on `region`, each measure still grain-correct): + +```sql +SELECT s.region, + t.ticket_count, + s.total_sales, + MEASURE(s.total_sales) / MEASURE(t.ticket_count) TO MEASURE AS sales_per_ticket +FROM tickets t JOIN sales s ON t.region = s.region; +``` + +### 7.7 Subqueries and CTEs + +Measures MUST propagate unchanged (type and grain preserved) through nested query structure: +subqueries (including a derived table in a `FROM` clause), scalar subqueries, CTEs, and views. A +measure defined in one relation can therefore be carried through and evaluated at the top, even +where it appears unevaluated in an intermediate select list. + +Whether a top-level query may return an unevaluated measure is deliberately left undefined +by this specification (Section 13.1); intermediate propagation is fully specified here. + +Non-normative example: + +```sql +SELECT region, MEASURE(total_revenue) +FROM (SELECT region, total_revenue FROM v WHERE order_date >= DATE '2025-01-01') t +GROUP BY region; -- total_revenue stays a measure in the subquery, evaluated at the top +``` + +### 7.8 Set operators and DISTINCT + +`UNION` / `UNION ALL` / `INTERSECT` / `EXCEPT` / `DISTINCT` compare or append whole rows, but an +unevaluated measure has no value to compare or append. Therefore: + +- A set operator or `DISTINCT` applied to an input relation that carries one or more + unevaluated measure columns MUST be rejected. +- The same operators are valid once measures have been evaluated to scalars (via + `MEASURE()` in an aggregated subquery) or projected away. For example, a `UNION ALL` of + two subqueries that each select only fields is valid. + +Non-normative examples: + +```sql +(SELECT region FROM v) UNION (SELECT region FROM w); -- ok: only fields +(SELECT total_revenue FROM v) UNION (SELECT total_revenue FROM w); -- rejected: unevaluated measures +``` + +--- + +## 8. Measure evaluation and guarantees + +This section gives the single normative definition of how a measure is evaluated (Section 8.1), +then states, as consequences of it, the guarantees a conforming engine provides (Sections +8.2-8.4). A conforming engine MUST provide these guarantees for every measure column in every +query. The consumer-side obligations these guarantees assume are collected in Section 8.5. + +### 8.1 Evaluation semantics + +A measure is evaluated only through `MEASURE()` in an aggregated query (Sections 6, 7.2). To +evaluate `MEASURE(m)` in a query grouped by fields `D` (possibly empty; an empty `D` is +the grand total), the query's joins and `WHERE` filters are applied first; then, within each +output group, and before `m`'s aggregate is applied: + +1. rows that share the same row identity for `m` are collapsed to one (removing duplication a + join introduced); and +2. rows that carry no row identity for `m` are dropped (removing null-padded rows an outer join + introduced). + +`m`'s aggregate is then evaluated over the surviving, distinct source rows of the group. If a +group has no surviving source rows for `m` (its filters removed them all, or no row in the group +carried `m`'s identity, as on the null-padded side of an outer join), then `m` takes the +aggregate's standard **empty-input value** in the host SQL dialect: `SUM` and `AVG` yield NULL, +`COUNT` yields 0, and so on, just as the aggregate would over an empty input in an ordinary +query. + +This collapsing and dropping is internal to evaluating `m`: it changes only the rows `m`'s +aggregate sees, never the rows or groups the query returns. Which groups appear is fixed by the +query's joins, filters, and grouping, as in standard SQL; a group in which every row was dropped +is kept and takes `m`'s empty-input value, rather than vanishing from the result. + +Each measure in a query is evaluated this way independently, at its own grain (row identity, +Section 3.2); an expression combining several `MEASURE()` calls applies its scalar operators to +their per-group results, following the host dialect's standard scalar and NULL semantics, +including its own handling of cases such as division by zero (Section 7.2). + +Step 1 is unambiguous because a measure's aggregate inputs (its argument expressions) are +functionally determined by its row identity (Section 3.2): all rows sharing an identity carry +the same input values. Section 9 gives one concrete realization of this definition; any +implementation MUST produce the same results. + +### 8.2 G1: Slice-and-dice responsiveness + +`MEASURE(m)` responds to `WHERE` and `GROUP BY` just as a standard SQL aggregate over `m`'s +definition would: a `WHERE` predicate restricts the input rows `m` sees (Section 7.3), and the +query's grouping sets the output grain at which `m` is reported (Section 7.4). This follows +directly from Section 8.1, where filters and grouping are applied ahead of `m`'s aggregate. It +is what makes filtering and drill-down behave predictably and identically to non-measure +aggregates. + +### 8.3 G2: Grain safety (no duplication) + +`MEASURE(m)` aggregates `m` at its own grain without duplication, regardless of the shape of the +enclosing query: any join type and cardinality, and any other measures or relations present. +This is the content of steps 1-2 of Section 8.1: each source row of `m` contributes to +`m`'s aggregate at most once per output group, however many times a join duplicated it, and +rows an outer join left without identity do not contribute at all. `MEASURE(m)` therefore does +not depend on the cardinality of any other relation or measure in the query. + +G2 structurally eliminates the two classic multi-table aggregation errors: + +- **Fan-out (double counting).** Joining a measure relation to a finer-grained table (one + source row -> many joined rows) does not inflate the measure; each source row is counted + once. +- **Chasm trap.** Joining two independent fact relations through a shared field does not + produce cross-product inflation; each measure aggregates at its own grain independently and + the results are combined afterward. + +### 8.4 Corollary: Closure and stable field domains + +Because `MEASURE()` behaves as an ordinary SQL aggregate (G1), a measure relation behaves as an +ordinary relation: the set of groups a query returns (its field domain) is fixed by its +grouping fields and filters, just as with standard `GROUP BY`, and does not depend on which +measures it selects. In particular, for a measure relation `R`, the set of field values returned +by + +```sql +SELECT d, MEASURE(m1) FROM R GROUP BY d +``` + +and by + +```sql +SELECT d, MEASURE(m2) FROM R GROUP BY d +``` + +MUST be the same (the domain of `d`), differing only in the measure column. A conforming +engine MUST preserve this closure property. + +> **Why this matters (non-normative).** SQL-native tools apply rewrites and optimizations that +> assume a relation's field domain does not shift when measures are added or removed, for +> example folding a Top-N filter subquery into the main query when both use the same grouping +> field. Those rewrites are sound against this interface because of the closure corollary, and +> are unsound against opaque multi-table interfaces whose field domains can vary with the +> measures in play. Preserving closure is therefore essential to P1 (a transparent relational +> interface). + +### 8.5 Consumer obligations + +The guarantees above hold when a consuming tool observes the following. Unlike G1-G2, these are +requirements on the *consumer*, not the engine. + +- **Evaluate measures only through `MEASURE()`.** A consumer references a measure by name and lets + the engine evaluate it (P2); it MUST NOT reconstruct a measure from the underlying columns, + which would forgo the grain guarantee. +- **Do not reaggregate a `MEASURE()` result on the client.** A `MEASURE()` value is already + aggregated at the query's grain; combining such values across groups on the client (for a + coarser grain, a drill-up, or a cached-then-refiltered view) is not valid, the same rule + that applies to non-decomposable aggregates such as `COUNT(DISTINCT ...)` or `MEDIAN`. To + change the grain, re-query the engine. For example, given per-state `MEASURE(revenue)`, a tool + MUST NOT sum those client-side for a national total; it re-queries grouped at the national + grain. + +Discovery of measures, model boundaries, and relationships (Section 11) is the remaining +consumer-side step; a tool that ignores the measure marker still reads its fields as ordinary +columns but will not know which columns are measures (Section 11.1). + +--- + +## 9. Reference evaluation strategy (non-normative) + +This section describes one strategy that realizes the evaluation semantics of Section 8.1 (and +thereby guarantees G1 and G2). It is non-normative: an engine MAY use any strategy (cardinality +analysis, aggregate-specific algebraic identities, preserving row identity implicitly in the +query structure, etc.), provided the results equal those of this reference +strategy. + +This realization makes row identity concrete as a **grain key**: a single key column `k` +synthesized when the measure is defined (for example from a declared key, or by row numbering). +`k` is non-null for every source row, unique per source row, and stable within a single query +execution. It is implicitly propagated through relational operators between the measure definition +and evaluation sites. + +Given a query grouping by fields `D` (possibly empty) with the joins and filters applied, a +measure whose aggregate is `agg(e1, ..., en)` over argument expressions `e1..en` is evaluated, +for each grouping row, by a correlated scalar subquery that de-duplicates to identity and then +aggregates: + +1. **De-duplicate to identity.** Take the joined, filtered rows of this group, discard those + where `k` is null (the identity-less rows of Section 8.1 step 2), and reduce to one row per + `k`, carrying each argument expression through with `ANY_VALUE` (well-defined because every + `ei` is functionally determined by `k`, Section 3.2). This leaves one row per distinct source + row of the measure in the group, removing join-introduced duplication. +2. **Aggregate.** Apply the aggregate over those de-duplicated rows. + +```sql +-- evaluated per grouping row `spine` of (SELECT DISTINCT D FROM joined): +SELECT agg(e1, ..., en) +FROM ( SELECT ANY_VALUE(e1) AS e1, ..., ANY_VALUE(en) AS en + FROM joined + WHERE k IS NOT NULL AND joined.D IS NOT DISTINCT FROM spine.D + GROUP BY k ) +``` + +Correlating each group to the joined rows on `D` (matched with `IS NOT DISTINCT FROM`, so a +NULL grouping key matches its own group) fixes the subquery to one value of `D`, so the aggregate +runs directly over that group's de-duplicated source rows. A group with no surviving source row +aggregates over the empty set, and so takes the aggregate's empty-input value (Section 8.1): +`NULL` for `SUM`, `0` for `COUNT`, rather than being dropped. `COUNT(*)` carries +nothing through step 1 (rows are still reduced to one per `k`) and counts the distinct source +rows; set-based aggregates behave as expected over them, so `COUNT(DISTINCT ei)`, `AVG(ei)`, or a +multi-argument `CORR(a, b)` each consume every distinct source row once. + +Each measure in the query is evaluated by its own such subquery at its own grain, and the query's +scalar expressions over the resulting per-group scalars are applied per group (Section 7.2). + +--- + +## 10. Multi-table models + +### 10.1 Shape + +A multi-table Ossie model is mapped to a set of relations (database tables, views, and measure +relations), each with its own grain, fields, and measures, together with a set of relationships +among them. +A tool discovers the relations and relationships (Section 11), then +generates queries that join them using its own SQL, reading measures via `MEASURE()`. + +Nothing in a tool's multi-table workflow needs to change: the tool joins measure relations +just as it joins any table, and the only substitution is `MEASURE(name)` in place of a raw +aggregate. The engine supplies G1 and G2 across the joins. + +Measure relations at peer grains (multi-fact models) are supported directly: a query MAY join +two or more measure relations and evaluate a measure from each; by G2 each is computed at its own +grain independently (Section 7.6). + +Because the engine guarantees each measure's grain, a consumer can enrich a model freely: join +a measure relation to any other table with ordinary SQL, or take a different join path, without +re-declaring the model. Every measure still evaluates at its own grain (G2), so a tool, +or an AI agent choosing its own path, never reaches a cliff where it must fall back to +hand-written, grain-safe aggregation. + +### 10.2 Consistency contract + +This is the cross-tool contract, and the boundary of what the standard fixes. + +- **Standardized: which rows each measure aggregates, given the same query.** Two conforming + engines evaluating the same query (same relations, joins, join types, filters, and grouping) + aggregate each measure over the same de-duplicated, identity-bearing source rows at the same + grain (G1, G2), and return exactly the rows standard SQL returns for that query; measures never + change which rows appear. What the interface does not standardize is the host dialect's own + evaluation of the underlying aggregate and scalar expressions over those rows. Like any SQL + extension, it inherits dialect differences in approximate aggregates, floating-point + accumulation, locale- or timezone-sensitive functions, and the like. Two engines agree on a + measure's value to the extent they agree on that underlying SQL; the interface adds no + divergence of its own. +- **Not standardized: which query a tool generates.** Which relations to include, which join + types to use (INNER vs. LEFT vs. FULL OUTER), and how to propagate filters are the tool's own + analytic choices. Different analyses genuinely call for different join and filter behavior, so + this is essential flexibility, not ambiguity, and this specification does not prescribe it. + +Consequently, two tools that render the "same" user intent as different queries MAY return +different rows, each correct for the question its query actually asks, and any row present in +both carries the same measure values (per the contract above). The standard fixes the result of a +query and leaves the choice of query to the consumer. + +Coming from a BI semantic layer, note the difference: here the query's `FROM` and joins are +authoritative. The engine evaluates each measure over the rows that query produces and never +adds, removes, or rewrites a join in response to which measures or fields are selected. Adding +or dropping a measure, or changing the grouping, changes only the projection or grouping, not the +joins. Regenerating the query as the selection changes, if wanted, is the consuming tool's job +(a Layer 3 concern, Section 1.1). + +> An engine or model author MAY publish advisory hints (a preferred join type, a filter +> propagation direction, a declared cardinality). Hints express intent and MAY inform a tool's +> choices, but they do not change the guarantees above and are specified outside this document. + +### 10.3 Home relations and Ossie metrics + +Every measure in this interface is a column of a relation (Section 3.1), evaluated at a +well-defined grain (row identity, Section 3.2). A measure whose correct evaluation depends on a +join is defined on a relation whose own definition includes that join, so its row identity +is well-defined at the point of definition. For example, a measure `SUM(qty * price)` reading +columns from both `orders` and `products` is a measure on a relation defined +`FROM orders JOIN products`, evaluated at that joined relation's row identity; the consumer +evaluates it with `MEASURE()` like any other. This is the general pattern: fold a measure's +required joins into its home relation's definition, rather than leaving them to the consuming +query. + +Multi-relation measures fit the same pattern. A measure derived from measures on more than one +relation (say a ratio of a measure from each of two facts, Section 5.2) lives on a home relation +whose field domain is fixed, such as a conformed dimension shared by the two facts. +Defined there as `(MEASURE(a) / MEASURE(b)) TO MEASURE`, it is an ordinary measure relation that +obeys closure (Section 8.4); Appendix A.7 works this through with a conformed-dimension bridge. + +A free-floating measure (attached to no relation, evaluated over whatever join path each query +happens to take) does not belong in Layer 2. Its domain and result shift from query to query. It +is a family of results rather than a single metric, and cannot carry the closure or cross-tool +consistency Layer 2 guarantees. (The multi-relation measures one might otherwise express with a +floating measure are the derived measures above, each defined on a home relation.) Such measures +could instead fit a Layer 3 wide-table interface, whose job is precisely to resolve a join path +per query (Sections 1.1, 10.2); that is outside the scope of this specification. + +**Recommendation to Ossie (Layer 2 needs per-relation measures).** Apache Ossie places +`metrics` at the model level, above datasets (Section 3.4); it has no notion of a metric +belonging to a dataset. To expose a model through this interface, every measure needs a home +relation. We therefore recommend that Ossie add **per-dataset (per-relation) measures** (a +metric attached to the dataset that supplies its grain), so a model's metrics map directly onto +queryable measure columns. A single-dataset metric attaches to its dataset; a metric requiring a +join attaches to a dataset whose definition includes that join, per the pattern above. Apache +Ossie [PR #343](https://github.com/apache/ossie/pull/343) proposes exactly this: dataset-scoped +metrics (`datasets[].metrics`) restricted to the dataset's own fields, with cross-dataset metrics +staying model-scoped. We recommend its adoption. + +--- + +## 11. Model discovery + +A tool needs to discover, through standard SQL metadata channels, (a) which columns are +measures, (b) which relations belong to a model, and (c) how the relations are related. This +interface reuses existing mechanisms wherever possible so that a tool needs minimal new +capabilities. + +### 11.1 Measure discovery + +Measure relations MUST be discoverable as ordinary relations through standard metadata +operations (`DESCRIBE`, `information_schema`, a zero-row projection). A conforming engine MUST +expose, through these channels, which columns are measures, for example by qualifying a +measure column's reported type with a `MEASURE` marker: + +``` +-- illustrative: DESCRIBE model.sales +column_name data_type +order_date DATE +region VARCHAR +order_count BIGINT MEASURE +total_revenue DECIMAL MEASURE +``` + +A tool that does not recognize the marker MUST still be able to read the relation's fields +as ordinary columns; it simply will not know which columns are measures. + +The type-name marker keeps measure status loud: a measure announces itself in the column type +every consumer already inspects, so it is hard to mistake a measure for a plain column, unlike a +separate metadata table, which a naive consumer can silently fail to consult. Its cross-driver +robustness is the open concern (JDBC/ODBC drivers normalize, alias, or truncate type-name +strings), and whether a metadata-relation fallback (for example +`information_schema.semantic_columns`) should be required alongside it is left open +(Section 13.1). + +### 11.2 Model boundary + +An engine SHOULD make it easy for a tool to find all relations of one model. In increasing +order of explicitness: + +- **Baseline (do nothing).** Users connect relations themselves. This MUST remain possible; + the interface MUST NOT break tools that build models by hand. +- **Schema-native grouping (RECOMMENDED).** A tool importing a model SHOULD place all of a + model's relations in a single schema, and consuming tools SHOULD give a schema preferential + treatment (for example a one-click "import all tables in this schema"). This needs no new + protocol. Its limitation is cross-schema membership. +- **Explicit model tagging (OPTIONAL).** A relation MAY declare model membership as a property + (illustratively `MODEL = 'sales'`), and the engine MAY expose a virtual schema per model. + This gives an enforceable boundary and handles cross-schema membership, at the cost of new + engine and tool capability. + +### 11.3 Relationship discovery + +Users can always define joins in their tool (the universal baseline), but a relation MAY also +carry **relationships** that make its intended, valid join paths discoverable and reusable +across tools. A relationship is a **named, directional, informational** declaration from a +**source** relation to a **target** relation, recording: + +- a **name**, unique among the source relation's relationships; +- the **target** relation; +- a **join predicate**: an equi-join on declared column pairs (`USING` columns, or `ON` + equalities between them), which is what current Ossie metadata represents; a relationship MAY + additionally use an arbitrary boolean `ON` expression, an extension beyond Ossie (see below); +- a **cardinality** (`ONE_TO_ONE`, `ONE_TO_MANY`, `MANY_TO_ONE`, `MANY_TO_MANY`): the expected + business cardinality induced by the join predicate, informational only and not enforced; and +- an optional **target alias**, required only to disambiguate a self-relationship. + +Relationships are directional and carry meaning: a `MANY_TO_ONE` relationship declares that the +target is, in effect, an attribute of the source. Two relations MAY be connected by more than +one relationship, each with a distinct name (for example a `flights` relation relating to +`cities` once as the `departure city` and once as the `arrival city`). + +A relationship is **not a foreign key.** It requires no key or uniqueness on the target, allows +several relationships over the same fields, and is purely informational, so it expresses +analytic relationships foreign keys cannot. + +**Discovery.** A relationship is surfaced through standard metadata as a table constraint of +type `RELATIONSHIP` on its **source** relation: it appears in `DESCRIBE` output and in +`information_schema.table_constraints` (`CONSTRAINT_TYPE = 'RELATIONSHIP'`), and is declared or +removed with ordinary `ADD` / `DROP CONSTRAINT` DDL. Returning relationships on the source, not +the target, keeps discovery from fanning in every fact that points at a shared dimension. + +```sql +-- illustrative +ALTER VIEW model.flights + ADD CONSTRAINT departure_city MANY_TO_ONE RELATIONSHIP TO model.cities + ON flights.departure_city_id = cities.id; +``` + +An Ossie `relationship` maps onto the equi-join baseline: its `from`/`to` datasets are the +source and target and its `from_columns`/`to_columns` the joined column pairs (`from` the many +side, `to` the one), which round-trips between this interface and the Ossie model. The richer +elements above (explicit cardinality, semantic roles, and arbitrary boolean `ON` predicates) +have no home in current Ossie relationship metadata and do not round-trip without a future Ossie +schema extension (Section 13.2). + +**Using a relationship.** A tool reads the relationships on a relation, selects the one that +fits the analysis, and generates the join from its predicate. The relationship fixes the +predicate, cardinality, and role; the tool still chooses the join type (INNER, LEFT, ...) and +how filters propagate (Section 10.2), and the join then behaves under Section 7.6 like any +other. ASOF and range relationships, and `RELY` optimization hints, are future extensions +(Section 13.2). + +### 11.4 Descriptive metadata (non-normative) + +Human- and AI-facing descriptors that have standard SQL-metadata homes (a field's or measure's +label / display name, its description / comment, and its synonyms) SHOULD be surfaced through the +engine's normal column and table metadata so tools can present them. Ossie annotations without +such a home (`ai_context` instructions and examples, and `custom_extensions`) are out of scope +for this interface (Section 13.2); an engine MAY surface them out-of-band. + +--- + +## 12. Conformance + +An implementation conforms to this specification if it satisfies all of the following: + +- **C1 (Type discipline).** It enforces the measure type rules R1 and R2 (Section 4) at + analysis time. +- **C2 (Operator semantics).** It implements the operator behaviors of Section 7 for measure + columns, including rejecting the constructs Section 7 requires be rejected. +- **C3 (G1).** It provides slice-and-dice responsiveness (Section 8.2). +- **C4 (G2).** It provides grain-safe, duplication-free aggregation for every measure in every + query (Section 8.3), realizing the evaluation semantics of Section 8.1, with results equal to + the reference strategy of Section 9. +- **C5 (Closure).** It preserves the closure / stable-field-domain corollary (Section 8.4). +- **C6 (Measure discovery).** Measure relations are discoverable through standard metadata + channels and measures are identifiable among their columns (Section 11.1). + +Discovery mechanisms beyond C6 (model boundary and relationship discovery, Sections 11.2-11.3) +are RECOMMENDED but not required for conformance. The SQL spelling used in examples (Sections +5, 6, 11.3) is illustrative and provisional; standardizing it is a goal (Section 1.3), so +conformance is defined on the semantics an implementation realizes, not yet on a fixed syntax. +C1-C6 are engine-side requirements; a consuming tool additionally observes the obligations of +Section 8.5. + +An implementation MAY additionally support Layer 1 as a portable expression language and Layer +3 as a wide-table interface; those are specified separately. + +--- + +## 13. Open questions and out of scope + +### 13.1 Open questions + +These are blocking: each must be resolved before a conforming implementation can deliver the +scope of Section 1.3. They are collected here rather than restated throughout; each section that +raises one points to it. + +- **Per-relation measures in Ossie.** Every measure in this interface needs a home relation + (Section 10.3), but Ossie's metrics are model-level and cannot supply one. Apache Ossie + [PR #343](https://github.com/apache/ossie/pull/343) proposes dataset-scoped metrics + (`datasets[].metrics`), exactly these per-relation measures, and we recommend the working + group adopt it; the conformance level remains to be settled. +- **Grain declaration.** A row identity can be derived automatically (for example by numbering + each source row), so the standard may not need a grain-declaration syntax at all. Whether + auto-derivation suffices, or an explicit declaration (e.g. a `PRIMARY KEY`) is needed when the + intended grain is coarser than one source row, is open (Section 3.2). +- **Top-level unevaluated measures.** Whether a top-level query may return a column of type + `T measure`. This is currently left to the engine, which MAY reject it or MAY offer a + compatibility mode (for example returning nulls of the base type); intermediate propagation + through subqueries is defined (Section 7.7). The working group should standardize one + behavior, since it affects cross-engine interop. +- **Exposure through standard database interfaces.** How the parts of an Ossie model with no + counterpart in a plain table (measures, relationships, and AI metadata) reach a tool + through ordinary JDBC/ODBC connectivity. The type-name `MEASURE` marker is preferred because it + is loud (Section 11.1), but its robustness across drivers, whether a metadata-relation fallback + (for example `information_schema.semantic_columns`) is required alongside it, and how + relationships (Section 11.3) and descriptive/AI metadata (Section 11.4) are surfaced the same + way, are open. + +### 13.2 Out of scope for this version + +These are not required by this version and do not block the scope above; each is a candidate for +a later extension. Where an item is a runtime behavior (for example grain and window controls), a +conforming engine MAY reject it or define its own behavior but MUST NOT rely on cross-engine +agreement; whether it is a feature or syntax, it is simply out of scope here. + +Deferred query and modeling features: + +- **Grain and window control semantics.** The detailed semantics of grain-manipulation and + windowing controls (Section 5.3) are out of scope, but may be defined by a later extension. +- **Parameterization.** Whether a model, a dataset, or a measure can take parameters (values + supplied at query time that vary within a single query) is out of scope. +- **Relationships beyond equi- and boolean joins.** The relationship model (Section 11.3) + covers `ON` and `USING` predicates. ASOF and range relationships, and `RELY` optimization + hints, are future extensions; whether Ossie standardizes the relationship declaration (name, + direction, cardinality, predicate, role) and at what conformance level is open. +- **Advisory hints.** Whether a model author's preferred join type or filter-propagation + direction is carried as discoverable hints (Sections 10.2, 11.3); part of Ossie's broader + hints effort. +- **Ossie annotations without a SQL-native home.** Ossie `ai_context` instructions and + examples, `custom_extensions`, and expression dialects other than the one selected for a + measure or field have no required representation in this interface and do not affect query + semantics; an engine MAY surface them out-of-band (synonyms, labels, and descriptions do have + a metadata home; Section 11.4). + +Deferred interface and interop concerns: + +- **Error model.** Standard error classes for the constructs this specification requires be + rejected (Sections 4, 7), so a tool can distinguish a type violation from an unsupported + feature from a transient failure. Currently engine-specific. +- **Interface version and capability negotiation.** How a tool discovers, at connection time, + that an endpoint implements this interface, at what version, and with which optional + capabilities (discovery relations, relationship metadata). +- **Conformance test suite.** A shared, portable suite that makes the conformance criteria + (Section 12) verifiable across engines rather than self-certified; this requires pinning + enough concrete syntax to write portable tests. +- **Adjacent concerns.** Row-level and column-level security interaction with measure + evaluation, and consumer result-cache reuse rules, are recognized but not addressed in this + version. + +--- + +## Appendix A. Worked examples (non-normative) + +These illustrate that ordinary tool-generated SQL, using only the `MEASURE()` substitution, +composes with the guarantees above. They assume a model with measure relations `orders` +(grain: order), `sales` (grain: sale), and `tickets` (grain: ticket). + +### A.1 Simple aggregation + +```sql +SELECT category, MEASURE(total_quantity) +FROM orders +GROUP BY category; +``` + +`total_quantity` is aggregated at the order grain and reported per category (G1). + +### A.2 Top-N filter via self-join + +A tool keeps the top 2 states by revenue by computing them in a subquery and joining back: + +```sql +SELECT o.category, MEASURE(o.total_quantity) +FROM orders o +JOIN ( + SELECT state_id, MEASURE(total_revenue) AS r + FROM orders + GROUP BY state_id + ORDER BY r DESC, state_id + LIMIT 2 +) t ON o.state_id = t.state_id +GROUP BY o.category; +``` + +The join filters rows before `MEASURE(o.total_quantity)` is evaluated (G1) and does not inflate +it (G2). Because the interface is standard relational SQL, the `state_id` values the +subquery ranks are exactly the `state_id` values it joins back on. The field domain is fixed by +the relation and its filters, independent of which measure is selected (closure, Section 8.4). A +wide-table interface whose per-query domains shift with the measures requested has no such +guarantee: the subquery's top two states need not match those in the outer join, so the filter +can silently keep the wrong rows. When the filtered and displayed fields coincide, closure also +lets a tool fold the subquery into the main query. + +### A.3 Level-of-detail two-stage aggregation + +Average, over states, of each state's total revenue: + +```sql +SELECT d.category, AVG(m.state_revenue) +FROM (SELECT category, state_id FROM orders GROUP BY category, state_id) d +JOIN (SELECT state_id, MEASURE(total_revenue) AS state_revenue + FROM orders GROUP BY state_id) m + ON d.state_id IS NOT DISTINCT FROM m.state_id +GROUP BY d.category; +``` + +The inner `MEASURE(total_revenue)` is grain-correct per state (G2); the outer `AVG` is an +ordinary aggregate over the already-evaluated scalar. + +### A.4 Multi-fact, cross-grain ratio + +```sql +SELECT s.region, + MEASURE(t.ticket_count), + MEASURE(s.total_sales), + MEASURE(s.total_sales) / MEASURE(t.ticket_count) AS sales_per_ticket +FROM tickets t JOIN sales s ON t.region = s.region +GROUP BY s.region; +``` + +The many-to-many join on `region` neither inflates `ticket_count` nor `total_sales`; each is +evaluated at its own grain (G2), and the ratio is applied per region afterward (7.2). + +### A.5 Fan-out: joining to a finer-grained table + +Revenue per category, with the fact joined to its line items (one order -> many items): + +```sql +SELECT o.category, MEASURE(o.total_revenue) +FROM orders o +JOIN order_items i ON i.order_id = o.order_id +GROUP BY o.category; +``` + +The join produces one row per order item, but `MEASURE(o.total_revenue)` de-duplicates back to +the order grain (G2): each order's revenue is counted once, not once per line item. + +### A.6 Join type: the tool's choice + +The same measures under an inner and a left join, differing only in the join type: + +```sql +-- inner: only regions with both sales and tickets +SELECT s.region, MEASURE(s.total_sales), MEASURE(t.ticket_count) +FROM sales s JOIN tickets t ON s.region = t.region +GROUP BY s.region; + +-- left: every sales region; ticket_count is 0 where a region has no tickets +SELECT s.region, MEASURE(s.total_sales), MEASURE(t.ticket_count) +FROM sales s LEFT JOIN tickets t ON s.region = t.region +GROUP BY s.region; +``` + +The join type is the tool's analytic choice, not something the interface prescribes (Section +10.2). Either way each measure is evaluated at its own grain (G2). The left join keeps a region +that has sales but no tickets; there `MEASURE(t.ticket_count)` has no source rows and takes its +empty-input value (0 for `COUNT`, Section 8.1), while the inner join omits that region +entirely. For any region in both results the `MEASURE()` values are identical: the standard +fixes measure values for a given query, and the join choice fixes which rows appear. + +### A.7 Multi-fact via a conformed-dimension bridge + +`sales` (grain: sale) and `tickets` (grain: ticket) are independent facts sharing a conformed +`regions` dimension, keyed by `region_id`. Per Section 10.3, a cross-fact measure needs a home +relation whose definition already includes the joins to both facts. So it is defined not on +the bare `regions` dimension but on a relation built from it. Here that relation is a +`region_metrics` view: + +```sql +-- illustrative: the home relation folds the joins to both facts into its own definition +CREATE VIEW region_metrics AS +SELECT r.region_id, -- field + r.region_name, -- field + (MEASURE(sales.total_sales) / MEASURE(tickets.ticket_count)) + TO MEASURE AS sales_per_ticket -- derived measure +FROM regions r +LEFT JOIN sales ON sales.region_id = r.region_id +LEFT JOIN tickets ON tickets.region_id = r.region_id; +``` + +`region_metrics` carrying `sales_per_ticket` is then an ordinary measure relation. A consumer +evaluates `MEASURE(sales_per_ticket)` and MAY join `region_metrics` to further tables with +ordinary SQL: the measure stays grain-safe (G2), its `region_id` domain is fixed (closure, +Section 8.4), and no model re-declaration is needed to bring in another table. + +This is how Layer 2 covers multi-fact metrics without free-floating measures (Section 10.3): +making the bridge a relation makes explicit two things a floating measure, left to a per-query +join path, would leave to inference: + +- **The join path that bridges the facts** is fixed by the home relation's definition, so a + different bridge (a different conformed dimension, or different join types) is a different, + separately declared measure, not a per-query guess. Two multi-fact measures that differ only in + how the bridge is built are two distinct, individually correct measures here, rather than one + ambiguous result. +- **A stable domain**: `sales_per_ticket` is defined for every region, with the empty-input fill + of Section 8.1 where a fact has no rows, not a domain that shifts with which measures a query + happens to select. + +--- + +## Appendix B. Glossary + +Apache Ossie terms (Section 3.4): + +- **Apache Ossie (Open Semantic Interchange).** The vendor-neutral semantic-model standard this + interface exposes; model spec version `0.2.0.dev0`. +- **Dataset (Ossie).** A logical table (fact or dimension) in an Ossie model; exposed here as a + *relation*. +- **Field (Ossie).** A scalar column of a dataset; our *field* (Ossie's term, adopted). +- **Metric (Ossie).** An aggregate expression, defined at the Ossie *model* level; exposed here + as a *measure* on a home relation (Section 10.3). +- **Relationship (Ossie).** A directional link between datasets (`from`/`to`, + `from_columns`/`to_columns`); `from` is the many side, `to` the one. Surfaced via the + relationship model of Section 11.3. + +Interface terms: + +- **Field.** An ordinary (non-measure) SQL column of a relation, of scalar type `T`. +- **Measure.** A column of type `T measure` carrying an unevaluated aggregate calculation and a + grain. +- **Measure relation.** A relation with at least one measure column. +- **Grain / row identity.** The identity of the source rows a measure aggregates, fixed at + definition: distinct source rows have distinct identities, and equal-identity rows are the + same source row. The **grain key** (Section 9) is one implementation of it. +- **`MEASURE(m)`.** The aggregate function that evaluates measure `m` at its grain, returning + `T`. +- **Base / derived measure.** A measure defined directly from an aggregate expression, versus + one defined from `MEASURE()` over other measures. +- **Cross-grain (multi-table) measure.** A derived measure referencing measures from more than + one relation. +- **G1 (slice-and-dice).** Measures respond to WHERE and GROUP BY like standard aggregates. +- **G2 (grain safety).** Measures aggregate at their grain without duplication regardless of + query shape. +- **Closure.** A measure relation behaves as a relation; its field domain is independent of + which measures are queried. +- **Fan-out / chasm trap.** The two classic duplication errors G2 eliminates. + +--- + +## Appendix C. Source material + +- J. Hyde and J. Fremlin. *Measures in SQL.* SIGMOD-Companion 2024. + https://arxiv.org/abs/2406.00251. The measure model this specification builds on: a measure + attaches a context-sensitive calculation to a table and expands transparently into ordinary SQL. +- Ossie proposal: [BI-Semantic Layer Interface](https://docs.google.com/document/d/1uKc3x9NlvV9O9UPdvlnCTDxsc-XWsQrxb0ylG45H-IQ/edit). + Transparent model with correctness-guaranteed measures; the interface shape adopted here. +- Ossie proposal: [Layered Query Interface](https://docs.google.com/document/d/1FOtRNBu6UqA2yeOwUa4jt1r2v45BWQJqS5_BTk34a1c/edit). + The Layer 1 / 2 / 3 framing of Section 1.1. From 8fd83bb65c836bf9685dc0d2ca872846f3791503 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Thu, 3 Sep 2026 12:27:08 -0700 Subject: [PATCH 2/8] [core-spec] Add Appendix D: relationship to Measures in SQL Syncs the relational_semantics.md spec with the upstream draft, adding Appendix D relating the design to Hyde and Fremlin's Measures in SQL. --- core-spec/relational_semantics.md | 60 +++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/core-spec/relational_semantics.md b/core-spec/relational_semantics.md index d04a3b1a4..3b4b6185c 100644 --- a/core-spec/relational_semantics.md +++ b/core-spec/relational_semantics.md @@ -1252,3 +1252,63 @@ Interface terms: Transparent model with correctness-guaranteed measures; the interface shape adopted here. - Ossie proposal: [Layered Query Interface](https://docs.google.com/document/d/1FOtRNBu6UqA2yeOwUa4jt1r2v45BWQJqS5_BTk34a1c/edit). The Layer 1 / 2 / 3 framing of Section 1.1. + +--- + +## Appendix D. Relationship to *Measures in SQL* + +This interface builds on the measure model of Hyde and Fremlin, +[*Measures in SQL*](https://arxiv.org/abs/2406.00251). It keeps that model's core: the `T measure` +column type, the transparent expansion of a measure into ordinary SQL, and closure (a relation +with measures behaves like any other relation). The rest of this appendix records where it +differs: naming and syntax, how a measure is evaluated, and some considerations for extending +the proposal later. + +### Naming and syntax + +The evaluation function is `MEASURE()`, not the paper's `AGGREGATE()`. `AGGREGATE` already names a +different function in some engines (in Databricks it is a higher-order function over arrays), and +the evaluation function needs a name unlikely to collide with existing functions or reserved words +across popular SQL engines. + +A measure is defined with `expr TO MEASURE AS name`, not the paper's `expr AS MEASURE name`. +`AS MEASURE` is cleaner, but ambiguous: in standard SQL `expr AS measure` already means "give this +column the alias `measure`." + +### Evaluating a measure + +A measure is evaluated only through `MEASURE()` (R2). The paper treats its `AGGREGATE()` wrapper +as largely cosmetic: in an aggregate query a bare measure reference is evaluated implicitly. We +require the explicit call, so the conversion of a `T measure` to a scalar `T` happens at one +visible place. Further, in SQL, an aggregate function in the `SELECT`, `HAVING`, or `ORDER BY` +clause is what turns a projection into an aggregation; a bare measure column that implicitly +aggregates breaks that expectation, so requiring `MEASURE()` keeps the query syntactically +ordinary SQL. + +`MEASURE()` evaluates a measure over exactly the rows the query selects, after its `WHERE` filters +and joins (Guarantee G1). In contrast, the paper defaults to not applying the query's `WHERE` or +joins to a measure; a consumer opts those in with `VISIBLE`. Applying the query's filters and +joins by default matches standard SQL behavior. + +### Extending the proposal + +Grain manipulation is explicitly out of scope in this version (Section 5.3), so this proposal does +not adopt the paper's `AT` operator or its consumer-modifiable *evaluation context* (a predicate +over a measure's dimension columns, adjusted at the call site with `ALL`, `SET`, or `WHERE`). A +measure here is anchored to its grain (Section 3.2) and evaluated only through `MEASURE()`. + +The functionality grain manipulation enables, such as percent of total and period over period, is +important, and we recommend strongly that it be considered as a follow-up extension. Two things +are worth considering in that design. + +**Encapsulation.** The paper encapsulates a measure's *formula*: a consumer uses a measure without +seeing its definition or the tables it reads (Principle P2). We believe encapsulation should cover a +measure's *meaning* as well: a measure has a single correct result for a given query, and a +consumer can slice, dice, and combine measures but cannot reach in and change what a measure +means. An extension should keep this property: a consumer evaluates a measure at additional +contexts and combines the results, without escaping its definition. + +**Column identity.** The paper's `SET` and `ALL` modifiers identify filters by dimension, which +requires deciding when two filters are on the same dimension. SQL has no general way to decide +that two columns are the same, so a context modifier for this interface will need its own answer +to that question. From 81cabe1b2d36b796872ce1bcfa7258a83533f953 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Tue, 13 Oct 2026 09:23:24 -0700 Subject: [PATCH 3/8] [core-spec] Type MEASURE()'s argument as an expression of type T measure Generalize the argument rule from "a measure column" to "an expression of type T measure". In this version the only such expression is a measure column reference, so behavior is unchanged, but later extensions (e.g. a measure with a modified evaluation context, or an anonymous inline measure) can be added without changing the rule. Co-authored-by: Isaac --- core-spec/relational_semantics.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/core-spec/relational_semantics.md b/core-spec/relational_semantics.md index 3b4b6185c..9be9d5c8f 100644 --- a/core-spec/relational_semantics.md +++ b/core-spec/relational_semantics.md @@ -371,9 +371,14 @@ A measure is evaluated through the aggregate function `MEASURE`: appear (e.g. SELECT list, HAVING, ORDER BY), and it MUST NOT appear where a standard aggregate may not. Placing it in a SELECT list or `HAVING` makes the query aggregated (Section 7.2), just as any aggregate does. -- **Argument.** The argument MUST resolve to a measure column (a column of type `T measure`), - optionally qualified by relation alias (`MEASURE(o.revenue)`). It MUST NOT be an arbitrary - scalar expression. +- **Argument.** The argument MUST be an expression of type `T measure`. In this version, the + only such expression is a reference to a measure column, optionally qualified by relation + alias (`MEASURE(o.revenue)`). Because no operator or function is defined on `T measure` (R1), + a scalar expression over measures, such as `MEASURE(a + b)`, is not well-typed. Write + `MEASURE(a) + MEASURE(b)` instead. + Later extensions may add other expressions of type `T measure`, such as a measure with a + modified evaluation context or an anonymous inline measure. This rule admits them without + change. `MEASURE(m)` evaluates `m`'s calculation at `m`'s grain, within the evaluation context established by the enclosing query's filters and grouping (Sections 7-8). From 4e494f32c0b0e25a54c6e38b7639bebd9e8481c4 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Tue, 13 Oct 2026 09:51:47 -0700 Subject: [PATCH 4/8] [core-spec] Define measure evaluation by input sets derived from source rows Rewrite Section 8.1 to define, for each output group, the set of a measure's source rows it aggregates, via derivation rules for every SQL operator between the measure's definition and its evaluation. No duplication (G2), filtering through intermediate queries, and outer-join null-padding all follow from the definition. Results are unchanged; the Section 9 reference strategy is one realization. Remove normative uses of "evaluation context" so the spec does not commit to a model for AT-style extensions. Co-authored-by: Isaac --- core-spec/relational_semantics.md | 121 ++++++++++++++++++------------ 1 file changed, 72 insertions(+), 49 deletions(-) diff --git a/core-spec/relational_semantics.md b/core-spec/relational_semantics.md index 9be9d5c8f..94cac4ef9 100644 --- a/core-spec/relational_semantics.md +++ b/core-spec/relational_semantics.md @@ -376,12 +376,11 @@ A measure is evaluated through the aggregate function `MEASURE`: alias (`MEASURE(o.revenue)`). Because no operator or function is defined on `T measure` (R1), a scalar expression over measures, such as `MEASURE(a + b)`, is not well-typed. Write `MEASURE(a) + MEASURE(b)` instead. - Later extensions may add other expressions of type `T measure`, such as a measure with a - modified evaluation context or an anonymous inline measure. This rule admits them without - change. + Later extensions may add other expressions of type `T measure`, such as a measure whose input + set is modified, or an anonymous inline measure. This rule admits them without change. -`MEASURE(m)` evaluates `m`'s calculation at `m`'s grain, within the evaluation context -established by the enclosing query's filters and grouping (Sections 7-8). +`MEASURE(m)` evaluates `m`'s calculation over its input set: the source rows of `m` that remain +in each group after the enclosing query's joins and filters (Section 8.1). > An engine MAY expose an additional spelling of this function (for example `AGG`) as a > synonym. Synonyms are non-normative. @@ -522,13 +521,13 @@ A join is logically applied before measures are evaluated. It affects a measure' in three ways, each handled by the evaluation semantics (Section 8.1) so the measure is still evaluated at its own grain: -- **Filtering.** Rows the join drops (an inner join's non-matches) are filtered from the - measure's input, just as a `WHERE` filter would, and that filtering is reflected in the measure. +- **Filtering.** Rows the join drops (an inner join's non-matches) are removed from the + measure's input set, just as a `WHERE` filter would remove them. - **Duplication.** A source row that matches several rows on the other side appears in several - result rows; the duplicates are collapsed before aggregation, so the measure is not inflated. -- **Null-extension.** An outer join's null-padded rows carry no row identity for the measure - (they correspond to no source row); they are dropped from this measure's aggregation input - (not from the query result; Section 8.1). + result rows, all deriving from the same source row; it enters the input set once, so the + measure is not inflated. +- **Null-extension.** An outer join's null-padded rows derive from no source row of the + measure, so they add nothing to its input set (they remain in the query result; Section 8.1). The result of a join of measure relations is itself a measure relation whose measures retain their original grains (Section 3.3), so joins compose. @@ -590,44 +589,68 @@ query. The consumer-side obligations these guarantees assume are collected in Se ### 8.1 Evaluation semantics -A measure is evaluated only through `MEASURE()` in an aggregated query (Sections 6, 7.2). To -evaluate `MEASURE(m)` in a query grouped by fields `D` (possibly empty; an empty `D` is -the grand total), the query's joins and `WHERE` filters are applied first; then, within each -output group, and before `m`'s aggregate is applied: - -1. rows that share the same row identity for `m` are collapsed to one (removing duplication a - join introduced); and -2. rows that carry no row identity for `m` are dropped (removing null-padded rows an outer join - introduced). - -`m`'s aggregate is then evaluated over the surviving, distinct source rows of the group. If a -group has no surviving source rows for `m` (its filters removed them all, or no row in the group -carried `m`'s identity, as on the null-padded side of an outer join), then `m` takes the -aggregate's standard **empty-input value** in the host SQL dialect: `SUM` and `AVG` yield NULL, -`COUNT` yields 0, and so on, just as the aggregate would over an empty input in an ordinary -query. - -This collapsing and dropping is internal to evaluating `m`: it changes only the rows `m`'s -aggregate sees, never the rows or groups the query returns. Which groups appear is fixed by the -query's joins, filters, and grouping, as in standard SQL; a group in which every row was dropped -is kept and takes `m`'s empty-input value, rather than vanishing from the result. - -Each measure in a query is evaluated this way independently, at its own grain (row identity, -Section 3.2); an expression combining several `MEASURE()` calls applies its scalar operators to -their per-group results, following the host dialect's standard scalar and NULL semantics, -including its own handling of cases such as division by zero (Section 7.2). - -Step 1 is unambiguous because a measure's aggregate inputs (its argument expressions) are -functionally determined by its row identity (Section 3.2): all rows sharing an identity carry -the same input values. Section 9 gives one concrete realization of this definition; any -implementation MUST produce the same results. +A measure is evaluated only through `MEASURE()` in an aggregated query (Sections 6, 7.2). Its +value in each output group is defined by the set of source rows it aggregates in that group. + +**Source rows.** A measure `m`'s **source rows** are the rows produced by the `FROM` clause in +effect where `m` is defined (Section 5.1). Each source row has a distinct row identity +(Section 3.2). + +**Derivation.** Every row of every relation that carries `m` *derives from* at most one of `m`'s +source rows: + +- Each row produced by `m`'s defining `FROM` clause derives from itself. +- Projection preserves derivation: an output row derives from the source row its input row + derives from. +- `WHERE` preserves derivation for the rows it keeps. A row it removes contributes nothing + downstream. +- A row produced by a join derives from the source row that the joined input row carrying `m` + derives from. When a join matches that input row several times, every resulting row derives + from the same source row. An outer join's null-padded rows on `m`'s side derive from no source + row. +- Subqueries, CTEs, and views preserve derivation (Section 7.7). + +These rules apply at every query level between `m`'s definition and its evaluation, including +the defining query and the evaluating query. Derivation is tracked separately for each measure +column, so in a self-join, each side's measure follows its own side's rows. + +**Input set.** In an aggregated query, the **input set** of `MEASURE(m)` for an output group `g` +is the set of `m`'s source rows from which at least one row of `g` derives. `m`'s aggregate is +evaluated over the input set, each source row contributing once, with its own argument values. +If the input set is empty (filters removed every source row, or no row of `g` derives from a +source row, as on the null-padded side of an outer join), `m` takes the aggregate's standard +**empty-input value** in the host SQL dialect: `SUM` and `AVG` yield NULL, `COUNT` yields 0, and +so on, just as the aggregate would over an empty input in an ordinary query. + +It follows that: + +- Filters and joins anywhere between `m`'s definition and its evaluation restrict `m`'s input + set (G1). +- A source row contributes to `m` at most once per group, however many rows derive from it, so + joins never inflate `m` (G2). +- Null-padded rows never contribute to `m`. + +The input set affects only the rows `m`'s aggregate sees, never the rows or groups the query +returns. Which groups appear is fixed by the query's joins, filters, and grouping, as in standard +SQL; a group whose input set is empty is kept and takes `m`'s empty-input value, rather than +vanishing from the result. + +Each measure in a query is evaluated this way independently, over its own input set. A derived +measure (Section 5.2) evaluates each measure it references over that measure's own input set +for the same group. An expression combining several `MEASURE()` calls applies its scalar +operators to their per-group results, following the host dialect's standard scalar and NULL +semantics, including its own handling of cases such as division by zero (Section 7.2). + +Section 9 gives one concrete realization of this definition; any implementation MUST produce the +same results. ### 8.2 G1: Slice-and-dice responsiveness `MEASURE(m)` responds to `WHERE` and `GROUP BY` just as a standard SQL aggregate over `m`'s definition would: a `WHERE` predicate restricts the input rows `m` sees (Section 7.3), and the query's grouping sets the output grain at which `m` is reported (Section 7.4). This follows -directly from Section 8.1, where filters and grouping are applied ahead of `m`'s aggregate. It +directly from Section 8.1: filters remove source rows from `m`'s input set, and each group's input +set contains only the source rows that the group's rows derive from. It is what makes filtering and drill-down behave predictably and identically to non-measure aggregates. @@ -635,9 +658,9 @@ aggregates. `MEASURE(m)` aggregates `m` at its own grain without duplication, regardless of the shape of the enclosing query: any join type and cardinality, and any other measures or relations present. -This is the content of steps 1-2 of Section 8.1: each source row of `m` contributes to -`m`'s aggregate at most once per output group, however many times a join duplicated it, and -rows an outer join left without identity do not contribute at all. `MEASURE(m)` therefore does +This follows from Section 8.1: `m`'s input set is a set of source rows, so each source row +contributes to `m`'s aggregate at most once per output group, however many times a join +duplicated it, and null-padded rows from an outer join do not contribute at all. `MEASURE(m)` therefore does not depend on the cardinality of any other relation or measure in the query. G2 structurally eliminates the two classic multi-table aggregation errors: @@ -720,7 +743,7 @@ for each grouping row, by a correlated scalar subquery that de-duplicates to ide aggregates: 1. **De-duplicate to identity.** Take the joined, filtered rows of this group, discard those - where `k` is null (the identity-less rows of Section 8.1 step 2), and reduce to one row per + where `k` is null (rows that derive from no source row, Section 8.1), and reduce to one row per `k`, carrying each argument expression through with `ANY_VALUE` (well-defined because every `ei` is functionally determined by `k`, Section 3.2). This leaves one row per distinct source row of the measure in the group, removing join-introduced duplication. @@ -1310,8 +1333,8 @@ are worth considering in that design. seeing its definition or the tables it reads (Principle P2). We believe encapsulation should cover a measure's *meaning* as well: a measure has a single correct result for a given query, and a consumer can slice, dice, and combine measures but cannot reach in and change what a measure -means. An extension should keep this property: a consumer evaluates a measure at additional -contexts and combines the results, without escaping its definition. +means. An extension should keep this property: a consumer evaluates a measure over modified +input sets and combines the results, without escaping its definition. **Column identity.** The paper's `SET` and `ALL` modifiers identify filters by dimension, which requires deciding when two filters are on the same dimension. SQL has no general way to decide From 036fede83ab3e5571cea2663dbd5de5985063767 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Tue, 13 Oct 2026 09:54:29 -0700 Subject: [PATCH 5/8] [core-spec] Replace "grain" with "row identity" throughout "Grain" was used for three different concepts: a measure's row identity, the level a query's grouping reports at, and AT-style manipulation of which rows a measure aggregates. Use "row identity" (and "source rows") for the first, "grouping"/"level" for the second, and "evaluation controls" for the third. Rename G2 to "No duplication" and Section 5.3 to "Evaluation controls". Add "input set" to the glossary. Co-authored-by: Isaac --- core-spec/relational_semantics.md | 284 +++++++++++++++--------------- 1 file changed, 144 insertions(+), 140 deletions(-) diff --git a/core-spec/relational_semantics.md b/core-spec/relational_semantics.md index 94cac4ef9..4dcf9f7da 100644 --- a/core-spec/relational_semantics.md +++ b/core-spec/relational_semantics.md @@ -67,10 +67,10 @@ alone gives flexibility, but not correctness. This specification proposes that BI and AI tools query Ossie models through standard SQL extended with correct measure semantics ([*Measures in SQL*](https://arxiv.org/abs/2406.00251)). The engine -evaluates a measure at its own grain, and it stays correct however the consumer filters, groups, -and joins. A tool evaluates it with a single `MEASURE()` aggregate and otherwise writes the SQL it -always would. Consumers get both governed, consistent measures and the flexibility of a SQL query -interface. +evaluates each measure over its own source rows, and it stays correct however the consumer +filters, groups, and joins. A tool evaluates it with a single `MEASURE()` aggregate and otherwise +writes the SQL it always would. Consumers get both governed, consistent measures and the +flexibility of a SQL query interface. ### 1.2 At a glance @@ -91,13 +91,13 @@ GROUP BY o.region ORDER BY MEASURE(o.total_revenue) DESC; ``` -`total_revenue` (on `orders`) and `ticket_count` (on `support`) are measures at different grains, -identified as measures from metadata (Section 11). `region`, `customer_id`, and `order_date` are -ordinary fields. The tool joins the two tables as it would any tables and evaluates a measure from -each with `MEASURE()`. The engine computes each measure at its own grain, so the many-to-many -join on `customer_id` inflates neither one (no fan-out or chasm trap, G2), while `WHERE` and -`GROUP BY` slice them like any aggregate (G1), on any field, join key or not. The same query -yields the same measure values across tools. A consuming tool writes `MEASURE(name)` where it +`total_revenue` (on `orders`) and `ticket_count` (on `support`) are measures over different source +rows, identified as measures from metadata (Section 11). `region`, `customer_id`, and `order_date` +are ordinary fields. The tool joins the two tables as it would any tables and evaluates a measure +from each with `MEASURE()`. The engine computes each measure over its own source rows, so the +many-to-many join on `customer_id` inflates neither one (no fan-out or chasm trap, G2), while +`WHERE` and `GROUP BY` slice them like any aggregate (G1), on any field, join key or not. The same +query yields the same measure values across tools. A consuming tool writes `MEASURE(name)` where it would otherwise write a raw aggregate, and generates SQL as it always has. Its obligations are in Section 8.5. @@ -116,8 +116,8 @@ with several tickets and vice versa. A raw `SUM`/`COUNT` over the joined rows wo This specification covers: -- The **data model**: relations carrying fields and measures, and the *grain* of a - measure (Section 3). +- The **data model**: relations carrying fields and measures, and the *row identity* of a + measure's source rows (Section 3). - The **type discipline** for measures (Section 4). - The **semantics** of defining measures and of querying them through SQL operators: projection, selection, joins, grouping, subqueries, and set operators (Sections 5-7). @@ -176,8 +176,8 @@ These principles motivate the normative rules and are themselves non-normative. This interface exposes an Ossie semantic model as SQL relations. Each dataset in an Ossie model becomes a relation, and the relation's columns are the dataset's fields and metrics. This section -defines that relational data model: relations of fields and measures (Section 3.1), the grain a -measure carries (Section 3.2), and how it maps to the Ossie model (Section 3.4). +defines that relational data model: relations of fields and measures (Section 3.1), the row +identity a measure carries (Section 3.2), and how it maps to the Ossie model (Section 3.4). ### 3.1 Relations, fields, measures @@ -186,7 +186,7 @@ SQL multiset semantics. Each column is exactly one of: - a **field**: an ordinary SQL column of some scalar type `T`; or - a **measure**: a column of type `T measure` (Section 4) that carries an unevaluated - aggregate calculation together with the grain at which it is to be evaluated. + aggregate calculation together with the row identity of the source rows it aggregates. A **measure relation** is a relation with at least one measure column. A relation with no measure columns is an ordinary relation and MUST behave exactly like a standard SQL relation @@ -195,12 +195,12 @@ under all operators; this specification imposes no additional requirements on it Fields and measures share a single column namespace within a relation. Column names within a relation MUST be unique. -### 3.2 Grain (row identity) +### 3.2 Row identity -Every measure carries the identity of the source rows it aggregates. This identity is its -**grain**. At the point where a measure is defined (Section 5), each row of the measure's source -has an identity. This **row identity** is remembered as the measure propagates through a query, -and it governs how the measure is aggregated (Section 8). +Every measure carries the identity of the source rows it aggregates. At the point where a +measure is defined (Section 5), each row of the measure's source has an identity. This **row +identity** is remembered as the measure propagates through a query, and it governs how the +measure is aggregated (Section 8). Two properties define row identity: @@ -210,7 +210,8 @@ Two properties define row identity: identity: a candidate identity under which two genuinely different rows collide is not a valid identity.) - A row **may lack an identity**. In a consuming query this happens to rows a join fabricates - that correspond to no source row (for example, the null-padded side of an outer join). + that derive from no source row (for example, the null-padded side of an outer join; Section + 8.1). A measure's row identity is fixed at definition time and is not altered by a consuming query. @@ -220,17 +221,18 @@ whether any representation is exposed to consumers, are discussed in Sections 9 row identity needs a declaration syntax at all is an open question (Section 13.1). It is a semantic notion and need not be a stored column; Section 9 gives one concrete realization. -> **Rationale (non-normative).** The grain is the single piece of state that distinguishes a +> **Rationale (non-normative).** Row identity is the single piece of state that distinguishes a > measure from a raw aggregate. A raw aggregate sums whatever rows reach it, so a join that > duplicates rows silently inflates downstream aggregates. A measure remembers the identity of -> its defining rows and so can be de-duplicated back to its own grain no matter how upstream -> operations multiply rows. +> its source rows and so aggregates each of them once, no matter how upstream operations +> multiply rows. -### 3.3 Multi-grain relations +### 3.3 Relations with measures from several sources -Distinct measures in the same relation MAY have distinct grains. A relation formed by joining -two measure relations (Section 7.6) carries measures at each contributing relation's original -grain. Each measure is evaluated at its own grain independently (Guarantee G2). +Distinct measures in the same relation MAY have distinct source rows. A relation formed by +joining two measure relations (Section 7.6) carries each contributing relation's measures with +their original row identities. Each measure is evaluated over its own source rows independently +(Guarantee G2). ### 3.4 Relationship to the Ossie semantic model @@ -243,15 +245,15 @@ names. The correspondence: | relation | `dataset` | A logical table (fact or dimension) exposed as a SQL relation. | | field | `field` | A scalar (non-measure) column; we adopt Ossie's term. Ossie's optional `dimension` role (e.g. `is_time`) is metadata this interface does not require. | | measure | `metric` | An aggregate expression on the model's data (of any scalar result type). Ossie currently places metrics at the *model* level, not on a dataset; Section 10.3 reconciles that and recommends Ossie add per-relation measures. | -| grain / row identity (Section 3.2) | `primary_key` | A dataset's declared primary key is a natural source of a measure's row identity. | +| row identity (Section 3.2) | `primary_key` | A dataset's declared primary key is a natural source of a measure's row identity. | | join relationship (Section 11.3) | `relationship` (`from`/`to`, `from_columns`/`to_columns`) | `from`/`to` map to source/target and the columns to the join predicate (`from` many, `to` one); Section 11.3 adds cardinality, roles, and arbitrary `ON`. | | Layer 1 expression | `expression` (per dialect) | Ossie carries SQL per dialect; a Layer 1 expression is the portable form (Section 1.1). | Ossie model metadata this interface does not natively surface (`ai_context`, `custom_extensions`, and non-selected expression dialects) is addressed under discovery (Section 11) and out of scope -(Section 13.2). A conforming engine MUST preserve metric -semantics: a metric evaluated via `MEASURE()` is computed from the model's metric definition (its -aggregate and grain), so two consumers issuing the same query agree on its value (Section 10.2). +(Section 13.2). A conforming engine MUST preserve metric semantics: a metric evaluated via +`MEASURE()` is computed from the model's metric definition (its aggregate and source rows), so two +consumers issuing the same query agree on its value (Section 10.2). --- @@ -298,7 +300,7 @@ including `MEASURE()`, is valid only in an aggregated context (R2). ### 4.3 Propagation Under projection, selection, joins, and subqueries, a measure column retains its type -`T measure` and its grain (Sections 3.2, 7). +`T measure` and its row identity (Sections 3.2, 7). --- @@ -324,9 +326,10 @@ Normatively: - The operand of a measure definition MUST be a valid SQL aggregate expression (in the Layer 1 expression language). A non-aggregate operand MUST be rejected. -- The measure's grain (Section 3.2) is captured from the `FROM` clause in effect at the point - of definition. The definition's own fields constrain nothing about the grain; the grain - is the identity of the source rows, not the projected fields. +- The measure's source rows, and their row identity (Section 3.2), are captured from the + `FROM` clause in effect at the point of definition. The definition's own fields constrain + nothing about the row identity; it is the identity of the source rows, not of the projected + fields. - The result column has type `T measure`, where `T` is the aggregate expressions's output type. ### 5.2 Derived measures @@ -343,20 +346,20 @@ avg_order_value := (MEASURE(total_revenue) / COUNT(DISTINCT order_id)) TO MEASUR `MEASURE(total_revenue)` reads the existing revenue measure and `COUNT(DISTINCT order_id)` counts orders from a raw field. A derived measure delegates to the measures it references: each -is evaluated at its own grain, and the surrounding scalar expression is applied after those +is evaluated over its own source rows, and the surrounding scalar expression is applied after those evaluations (see Section 7.2). Its inner `MEASURE()` calls are captured unevaluated, just as `TO MEASURE` captures `SUM(x)`: they are evaluated at query time, in the consuming query's context (not when the derived measure is defined), so no `MEASURE()` is evaluated outside an aggregated context (R2). Derived measures MAY reference measures from more than one relation; -such a measure is a **multi-table (cross-grain) measure** (Section 10.3). +such a measure is a **multi-table measure** (Section 10.3). -### 5.3 Grain and window controls +### 5.3 Evaluation controls -Because a measure carries its own grain, an engine MAY offer controls that manipulate that -grain: including or excluding fields from the evaluation grain, semi-additive selection, and -windowing (cumulative, trailing, period-over-period). These controls are extensions layered on -top of the base semantics defined here; their specification is out of scope for this document. -When present, they MUST preserve the guarantees of Section 8 for the grain they declare. +An engine MAY offer controls that change which source rows a measure aggregates in each group: +including or excluding fields from the grouping a measure responds to, semi-additive selection, +and windowing (cumulative, trailing, period-over-period). These controls are extensions layered +on top of the base semantics defined here; their specification is out of scope for this +document. When present, they MUST preserve the no-duplication guarantee of Section 8. --- @@ -404,9 +407,9 @@ follow from the definitions of a measure column and the `MEASURE()` function: column, with no special rules thereafter. Every operator then behaves as in standard SQL. The only measure-specific behavior is that each -`MEASURE()` is evaluated at its measure's own grain (Section 8.1). That is what makes filtering, -grouping, and joining compose correctly (G1, G2). The subsections state only that grain behavior -and the few outright exceptions; the rest is standard SQL and is not restated. +`MEASURE()` is evaluated over its measure's own input set (Section 8.1). That is what makes +filtering, grouping, and joining compose correctly (G1, G2). The subsections state only that +behavior and the few outright exceptions; the rest is standard SQL and is not restated. The following table summarizes; the subsections are normative. @@ -417,8 +420,8 @@ The following table summarizes; the subsections are normative. | WHERE (7.3) | Ordinary predicate; filters the measure's input rows before evaluation. `MEASURE()` only via a scalar subquery. | | GROUP BY / GROUPING SETS / CUBE / ROLLUP (7.4) | Grouping ranges over fields; grouping is not defined over `T measure`. `GROUP BY ALL` groups by the non-measure select expressions. | | HAVING / ORDER BY (7.5) | `MEASURE()` allowed, as for any aggregate. | -| JOIN (7.6) | Any join type/condition over fields. Each measure retains its own grain and is de-duplicated before evaluation (G2). Joining on a measure is an error. | -| Subquery / CTE (7.7) | Measures propagate as `T measure` with grain intact. | +| JOIN (7.6) | Any join type/condition over fields. Each measure retains its own row identity and is de-duplicated before evaluation (G2). Joining on a measure is an error. | +| Subquery / CTE (7.7) | Measures propagate as `T measure` with row identity intact. | | Set operators, DISTINCT (7.8) | Error when an input carries an unevaluated measure. | ### 7.1 SELECT (unaggregated) @@ -426,7 +429,7 @@ The following table summarizes; the subsections are normative. In a SELECT without aggregation, a measure column is treated like any other column: - if projected (by name, renamed, or via `*`), it passes through, keeping its `T measure` - type and its grain; if not projected, it is dropped; + type and its row identity; if not projected, it is dropped; - it MAY be defined or derived by a measure definition in the projection (Section 5); - it MUST NOT be the operand of a scalar operator or function (R1); - it is not evaluated to a value here: `MEASURE()` is the only way to evaluate a measure, and a @@ -450,10 +453,10 @@ aggregate expression in its select list. In an aggregated SELECT: select list of an aggregated query MUST be rejected. - `MEASURE(m)` returns a scalar of type `T`; it MAY be combined with other scalars and aggregates in the select list, HAVING, and ORDER BY. -- For a derived measure or an expression combining several `MEASURE()` calls, each - `MEASURE()` is evaluated independently at its own grain and the surrounding scalar expression - applied per output group (Section 8.1). For example, `MEASURE(revenue) / MEASURE(ticket_count)` - divides two independently grain-correct aggregates within each group. +- For a derived measure or an expression combining several `MEASURE()` calls, each `MEASURE()` is + evaluated independently over its own input set and the surrounding scalar expression applied per + output group (Section 8.1). For example, `MEASURE(revenue) / MEASURE(ticket_count)` divides two + independently duplication-free aggregates within each group. Non-normative examples: @@ -486,7 +489,7 @@ WHERE (SELECT MEASURE(gross) FROM budget) > 100 GROUP BY region; -- ok: MEASURE ### 7.4 GROUP BY - Grouping (`GROUP BY`, `GROUPING SETS`, `CUBE`, `ROLLUP`) ranges over fields; grouping is not - defined over `T measure` (R1). The grouping sets the grain at which each measure is reported, + defined over `T measure` (R1). The grouping sets the level at which each measure is reported, the other half of Guarantee G1 (Section 8.2). - Under `GROUP BY ALL`, measures are not grouping columns: a bare (unevaluated) measure in the select list has nothing to group by and MUST be rejected; otherwise `ALL` groups by the @@ -519,7 +522,7 @@ OUTER, CROSS) and any join condition over fields (R1). A join is logically applied before measures are evaluated. It affects a measure's input rows in three ways, each handled by the evaluation semantics (Section 8.1) so the measure is still -evaluated at its own grain: +evaluated over its own source rows: - **Filtering.** Rows the join drops (an inner join's non-matches) are removed from the measure's input set, just as a `WHERE` filter would remove them. @@ -530,9 +533,9 @@ evaluated at its own grain: measure, so they add nothing to its input set (they remain in the query result; Section 8.1). The result of a join of measure relations is itself a measure relation whose measures retain -their original grains (Section 3.3), so joins compose. +their original row identities (Section 3.3), so joins compose. -Non-normative example (many-to-many join on `region`, each measure still grain-correct): +Non-normative example (many-to-many join on `region`, each measure still duplication-free): ```sql SELECT s.region, @@ -544,7 +547,7 @@ FROM tickets t JOIN sales s ON t.region = s.region; ### 7.7 Subqueries and CTEs -Measures MUST propagate unchanged (type and grain preserved) through nested query structure: +Measures MUST propagate unchanged (type and row identity preserved) through nested query structure: subqueries (including a derived table in a `FROM` clause), scalar subqueries, CTEs, and views. A measure defined in one relation can therefore be carried through and evaluated at the top, even where it appears unevaluated in an intermediate select list. @@ -648,19 +651,19 @@ same results. `MEASURE(m)` responds to `WHERE` and `GROUP BY` just as a standard SQL aggregate over `m`'s definition would: a `WHERE` predicate restricts the input rows `m` sees (Section 7.3), and the -query's grouping sets the output grain at which `m` is reported (Section 7.4). This follows +query's grouping sets the level at which `m` is reported (Section 7.4). This follows directly from Section 8.1: filters remove source rows from `m`'s input set, and each group's input set contains only the source rows that the group's rows derive from. It is what makes filtering and drill-down behave predictably and identically to non-measure aggregates. -### 8.3 G2: Grain safety (no duplication) +### 8.3 G2: No duplication -`MEASURE(m)` aggregates `m` at its own grain without duplication, regardless of the shape of the -enclosing query: any join type and cardinality, and any other measures or relations present. +`MEASURE(m)` aggregates `m` over its own source rows without duplication, regardless of the shape of +the enclosing query: any join type and cardinality, and any other measures or relations present. This follows from Section 8.1: `m`'s input set is a set of source rows, so each source row -contributes to `m`'s aggregate at most once per output group, however many times a join -duplicated it, and null-padded rows from an outer join do not contribute at all. `MEASURE(m)` therefore does +contributes to `m`'s aggregate at most once per output group, however many times a join duplicated +it, and null-padded rows from an outer join do not contribute at all. `MEASURE(m)` therefore does not depend on the cardinality of any other relation or measure in the query. G2 structurally eliminates the two classic multi-table aggregation errors: @@ -668,9 +671,9 @@ G2 structurally eliminates the two classic multi-table aggregation errors: - **Fan-out (double counting).** Joining a measure relation to a finer-grained table (one source row -> many joined rows) does not inflate the measure; each source row is counted once. -- **Chasm trap.** Joining two independent fact relations through a shared field does not - produce cross-product inflation; each measure aggregates at its own grain independently and - the results are combined afterward. +- **Chasm trap.** Joining two independent fact relations through a shared field does not produce + cross-product inflation; each measure aggregates over its own source rows independently and the + results are combined afterward. ### 8.4 Corollary: Closure and stable field domains @@ -708,14 +711,14 @@ requirements on the *consumer*, not the engine. - **Evaluate measures only through `MEASURE()`.** A consumer references a measure by name and lets the engine evaluate it (P2); it MUST NOT reconstruct a measure from the underlying columns, - which would forgo the grain guarantee. + which would forgo the no-duplication guarantee. - **Do not reaggregate a `MEASURE()` result on the client.** A `MEASURE()` value is already - aggregated at the query's grain; combining such values across groups on the client (for a - coarser grain, a drill-up, or a cached-then-refiltered view) is not valid, the same rule + aggregated at the query's grouping; combining such values across groups on the client (for a + coarser grouping, a drill-up, or a cached-then-refiltered view) is not valid, the same rule that applies to non-decomposable aggregates such as `COUNT(DISTINCT ...)` or `MEDIAN`. To - change the grain, re-query the engine. For example, given per-state `MEASURE(revenue)`, a tool - MUST NOT sum those client-side for a national total; it re-queries grouped at the national - grain. + change the grouping, re-query the engine. For example, given per-state `MEASURE(revenue)`, a + tool MUST NOT sum those client-side for a national total; it re-queries grouped at the national + level. Discovery of measures, model boundaries, and relationships (Section 11) is the remaining consumer-side step; a tool that ignores the measure marker still reads its fields as ordinary @@ -731,7 +734,7 @@ analysis, aggregate-specific algebraic identities, preserving row identity impli query structure, etc.), provided the results equal those of this reference strategy. -This realization makes row identity concrete as a **grain key**: a single key column `k` +This realization makes row identity concrete as a **row-identity key**: a single key column `k` synthesized when the measure is defined (for example from a declared key, or by row numbering). `k` is non-null for every source row, unique per source row, and stable within a single query execution. It is implicitly propagated through relational operators between the measure definition @@ -767,8 +770,8 @@ nothing through step 1 (rows are still reduced to one per `k`) and counts the di rows; set-based aggregates behave as expected over them, so `COUNT(DISTINCT ei)`, `AVG(ei)`, or a multi-argument `CORR(a, b)` each consume every distinct source row once. -Each measure in the query is evaluated by its own such subquery at its own grain, and the query's -scalar expressions over the resulting per-group scalars are applied per group (Section 7.2). +Each measure in the query is evaluated by its own such subquery over its own source rows, and the +query's scalar expressions over the resulting per-group scalars are applied per group (Section 7.2). --- @@ -777,7 +780,7 @@ scalar expressions over the resulting per-group scalars are applied per group (S ### 10.1 Shape A multi-table Ossie model is mapped to a set of relations (database tables, views, and measure -relations), each with its own grain, fields, and measures, together with a set of relationships +relations), each with its own fields and measures, together with a set of relationships among them. A tool discovers the relations and relationships (Section 11), then generates queries that join them using its own SQL, reading measures via `MEASURE()`. @@ -786,30 +789,29 @@ Nothing in a tool's multi-table workflow needs to change: the tool joins measure just as it joins any table, and the only substitution is `MEASURE(name)` in place of a raw aggregate. The engine supplies G1 and G2 across the joins. -Measure relations at peer grains (multi-fact models) are supported directly: a query MAY join -two or more measure relations and evaluate a measure from each; by G2 each is computed at its own -grain independently (Section 7.6). +Independent measure relations (multi-fact models) are supported directly: a query MAY join +two or more measure relations and evaluate a measure from each; by G2 each is computed over its +own source rows independently (Section 7.6). -Because the engine guarantees each measure's grain, a consumer can enrich a model freely: join -a measure relation to any other table with ordinary SQL, or take a different join path, without -re-declaring the model. Every measure still evaluates at its own grain (G2), so a tool, -or an AI agent choosing its own path, never reaches a cliff where it must fall back to -hand-written, grain-safe aggregation. +Because the engine guarantees that each measure is evaluated without duplication, a consumer can +enrich a model freely: join a measure relation to any other table with ordinary SQL, or take a +different join path, without re-declaring the model. Every measure still evaluates over its own +source rows (G2), so a tool, or an AI agent choosing its own path, never reaches a cliff where +it must fall back to hand-written, duplication-free aggregation. ### 10.2 Consistency contract This is the cross-tool contract, and the boundary of what the standard fixes. -- **Standardized: which rows each measure aggregates, given the same query.** Two conforming - engines evaluating the same query (same relations, joins, join types, filters, and grouping) - aggregate each measure over the same de-duplicated, identity-bearing source rows at the same - grain (G1, G2), and return exactly the rows standard SQL returns for that query; measures never - change which rows appear. What the interface does not standardize is the host dialect's own - evaluation of the underlying aggregate and scalar expressions over those rows. Like any SQL - extension, it inherits dialect differences in approximate aggregates, floating-point - accumulation, locale- or timezone-sensitive functions, and the like. Two engines agree on a - measure's value to the extent they agree on that underlying SQL; the interface adds no - divergence of its own. +- **Standardized: which rows each measure aggregates, given the same query.** Two conforming engines + evaluating the same query (same relations, joins, join types, filters, and grouping) aggregate + each measure over the same input set of source rows (Section 8.1; G1, G2), and return exactly the + rows standard SQL returns for that query; measures never change which rows appear. What the + interface does not standardize is the host dialect's own evaluation of the underlying aggregate + and scalar expressions over those rows. Like any SQL extension, it inherits dialect differences in + approximate aggregates, floating-point accumulation, locale- or timezone-sensitive functions, and + the like. Two engines agree on a measure's value to the extent they agree on that underlying SQL; + the interface adds no divergence of its own. - **Not standardized: which query a tool generates.** Which relations to include, which join types to use (INNER vs. LEFT vs. FULL OUTER), and how to propagate filters are the tool's own analytic choices. Different analyses genuinely call for different join and filter behavior, so @@ -833,8 +835,8 @@ joins. Regenerating the query as the selection changes, if wanted, is the consum ### 10.3 Home relations and Ossie metrics -Every measure in this interface is a column of a relation (Section 3.1), evaluated at a -well-defined grain (row identity, Section 3.2). A measure whose correct evaluation depends on a +Every measure in this interface is a column of a relation (Section 3.1), evaluated over source +rows with a well-defined row identity (Section 3.2). A measure whose correct evaluation depends on a join is defined on a relation whose own definition includes that join, so its row identity is well-defined at the point of definition. For example, a measure `SUM(qty * price)` reading columns from both `orders` and `products` is a measure on a relation defined @@ -861,10 +863,10 @@ per query (Sections 1.1, 10.2); that is outside the scope of this specification. `metrics` at the model level, above datasets (Section 3.4); it has no notion of a metric belonging to a dataset. To expose a model through this interface, every measure needs a home relation. We therefore recommend that Ossie add **per-dataset (per-relation) measures** (a -metric attached to the dataset that supplies its grain), so a model's metrics map directly onto -queryable measure columns. A single-dataset metric attaches to its dataset; a metric requiring a -join attaches to a dataset whose definition includes that join, per the pattern above. Apache -Ossie [PR #343](https://github.com/apache/ossie/pull/343) proposes exactly this: dataset-scoped +metric attached to the dataset that supplies its source rows), so a model's metrics map directly +onto queryable measure columns. A single-dataset metric attaches to its dataset; a metric +requiring a join attaches to a dataset whose definition includes that join, per the pattern +above. Apache Ossie [PR #343](https://github.com/apache/ossie/pull/343) proposes exactly this: dataset-scoped metrics (`datasets[].metrics`) restricted to the dataset's own fields, with cross-dataset metrics staying model-scoped. We recommend its adoption. @@ -991,7 +993,7 @@ An implementation conforms to this specification if it satisfies all of the foll - **C2 (Operator semantics).** It implements the operator behaviors of Section 7 for measure columns, including rejecting the constructs Section 7 requires be rejected. - **C3 (G1).** It provides slice-and-dice responsiveness (Section 8.2). -- **C4 (G2).** It provides grain-safe, duplication-free aggregation for every measure in every +- **C4 (G2).** It provides duplication-free aggregation for every measure in every query (Section 8.3), realizing the evaluation semantics of Section 8.1, with results equal to the reference strategy of Section 9. - **C5 (Closure).** It preserves the closure / stable-field-domain corollary (Section 8.4). @@ -1023,10 +1025,10 @@ raises one points to it. [PR #343](https://github.com/apache/ossie/pull/343) proposes dataset-scoped metrics (`datasets[].metrics`), exactly these per-relation measures, and we recommend the working group adopt it; the conformance level remains to be settled. -- **Grain declaration.** A row identity can be derived automatically (for example by numbering - each source row), so the standard may not need a grain-declaration syntax at all. Whether +- **Row identity declaration.** A row identity can be derived automatically (for example by + numbering each source row), so the standard may not need a declaration syntax at all. Whether auto-derivation suffices, or an explicit declaration (e.g. a `PRIMARY KEY`) is needed when the - intended grain is coarser than one source row, is open (Section 3.2). + intended identity is coarser than one source row, is open (Section 3.2). - **Top-level unevaluated measures.** Whether a top-level query may return a column of type `T measure`. This is currently left to the engine, which MAY reject it or MAY offer a compatibility mode (for example returning nulls of the base type); intermediate propagation @@ -1043,14 +1045,14 @@ raises one points to it. ### 13.2 Out of scope for this version These are not required by this version and do not block the scope above; each is a candidate for -a later extension. Where an item is a runtime behavior (for example grain and window controls), a +a later extension. Where an item is a runtime behavior (for example evaluation controls), a conforming engine MAY reject it or define its own behavior but MUST NOT rely on cross-engine agreement; whether it is a feature or syntax, it is simply out of scope here. Deferred query and modeling features: -- **Grain and window control semantics.** The detailed semantics of grain-manipulation and - windowing controls (Section 5.3) are out of scope, but may be defined by a later extension. +- **Evaluation control semantics.** The detailed semantics of evaluation controls, such as + totals and windowing (Section 5.3), are out of scope, but may be defined by a later extension. - **Parameterization.** Whether a model, a dataset, or a measure can take parameters (values supplied at query time that vary within a single query) is out of scope. - **Relationships beyond equi- and boolean joins.** The relationship model (Section 11.3) @@ -1087,7 +1089,7 @@ Deferred interface and interop concerns: These illustrate that ordinary tool-generated SQL, using only the `MEASURE()` substitution, composes with the guarantees above. They assume a model with measure relations `orders` -(grain: order), `sales` (grain: sale), and `tickets` (grain: ticket). +(one row per order), `sales` (one row per sale), and `tickets` (one row per ticket). ### A.1 Simple aggregation @@ -1097,7 +1099,7 @@ FROM orders GROUP BY category; ``` -`total_quantity` is aggregated at the order grain and reported per category (G1). +`total_quantity` is aggregated over orders and reported per category (G1). ### A.2 Top-N filter via self-join @@ -1138,10 +1140,10 @@ JOIN (SELECT state_id, MEASURE(total_revenue) AS state_revenue GROUP BY d.category; ``` -The inner `MEASURE(total_revenue)` is grain-correct per state (G2); the outer `AVG` is an +The inner `MEASURE(total_revenue)` is duplication-free per state (G2); the outer `AVG` is an ordinary aggregate over the already-evaluated scalar. -### A.4 Multi-fact, cross-grain ratio +### A.4 Multi-fact ratio ```sql SELECT s.region, @@ -1153,7 +1155,7 @@ GROUP BY s.region; ``` The many-to-many join on `region` neither inflates `ticket_count` nor `total_sales`; each is -evaluated at its own grain (G2), and the ratio is applied per region afterward (7.2). +evaluated over its own source rows (G2), and the ratio is applied per region afterward (7.2). ### A.5 Fan-out: joining to a finer-grained table @@ -1166,8 +1168,8 @@ JOIN order_items i ON i.order_id = o.order_id GROUP BY o.category; ``` -The join produces one row per order item, but `MEASURE(o.total_revenue)` de-duplicates back to -the order grain (G2): each order's revenue is counted once, not once per line item. +The join produces one row per order item, but `MEASURE(o.total_revenue)` aggregates each order +once (G2): each order's revenue is counted once, not once per line item. ### A.6 Join type: the tool's choice @@ -1185,20 +1187,20 @@ FROM sales s LEFT JOIN tickets t ON s.region = t.region GROUP BY s.region; ``` -The join type is the tool's analytic choice, not something the interface prescribes (Section -10.2). Either way each measure is evaluated at its own grain (G2). The left join keeps a region +The join type is the tool's analytic choice, not something the interface prescribes (Section 10.2). +Either way each measure is evaluated over its own source rows (G2). The left join keeps a region that has sales but no tickets; there `MEASURE(t.ticket_count)` has no source rows and takes its -empty-input value (0 for `COUNT`, Section 8.1), while the inner join omits that region -entirely. For any region in both results the `MEASURE()` values are identical: the standard -fixes measure values for a given query, and the join choice fixes which rows appear. +empty-input value (0 for `COUNT`, Section 8.1), while the inner join omits that region entirely. For +any region in both results the `MEASURE()` values are identical: the standard fixes measure values +for a given query, and the join choice fixes which rows appear. ### A.7 Multi-fact via a conformed-dimension bridge -`sales` (grain: sale) and `tickets` (grain: ticket) are independent facts sharing a conformed -`regions` dimension, keyed by `region_id`. Per Section 10.3, a cross-fact measure needs a home -relation whose definition already includes the joins to both facts. So it is defined not on -the bare `regions` dimension but on a relation built from it. Here that relation is a -`region_metrics` view: +`sales` (one row per sale) and `tickets` (one row per ticket) are independent facts sharing a +conformed `regions` dimension, keyed by `region_id`. Per Section 10.3, a cross-fact measure needs a +home relation whose definition already includes the joins to both facts. So it is defined not on the +bare `regions` dimension but on a relation built from it. Here that relation is a `region_metrics` +view: ```sql -- illustrative: the home relation folds the joins to both facts into its own definition @@ -1214,7 +1216,7 @@ LEFT JOIN tickets ON tickets.region_id = r.region_id; `region_metrics` carrying `sales_per_ticket` is then an ordinary measure relation. A consumer evaluates `MEASURE(sales_per_ticket)` and MAY join `region_metrics` to further tables with -ordinary SQL: the measure stays grain-safe (G2), its `region_id` domain is fixed (closure, +ordinary SQL: the measure stays duplication-free (G2), its `region_id` domain is fixed (closure, Section 8.4), and no model re-declaration is needed to bring in another table. This is how Layer 2 covers multi-fact metrics without free-floating measures (Section 10.3): @@ -1250,21 +1252,23 @@ Apache Ossie terms (Section 3.4): Interface terms: - **Field.** An ordinary (non-measure) SQL column of a relation, of scalar type `T`. -- **Measure.** A column of type `T measure` carrying an unevaluated aggregate calculation and a - grain. +- **Measure.** A column of type `T measure` carrying an unevaluated aggregate calculation and + the row identity of its source rows. - **Measure relation.** A relation with at least one measure column. -- **Grain / row identity.** The identity of the source rows a measure aggregates, fixed at +- **Row identity.** The identity of the source rows a measure aggregates, fixed at definition: distinct source rows have distinct identities, and equal-identity rows are the - same source row. The **grain key** (Section 9) is one implementation of it. -- **`MEASURE(m)`.** The aggregate function that evaluates measure `m` at its grain, returning - `T`. + same source row. The **row-identity key** (Section 9) is one implementation of it. +- **Input set.** For a measure and an output group, the set of the measure's source rows that + the group's rows derive from; the measure's aggregate is evaluated over it (Section 8.1). +- **`MEASURE(m)`.** The aggregate function that evaluates measure `m` over its input set, + returning `T`. - **Base / derived measure.** A measure defined directly from an aggregate expression, versus one defined from `MEASURE()` over other measures. -- **Cross-grain (multi-table) measure.** A derived measure referencing measures from more than +- **Multi-table measure.** A derived measure referencing measures from more than one relation. - **G1 (slice-and-dice).** Measures respond to WHERE and GROUP BY like standard aggregates. -- **G2 (grain safety).** Measures aggregate at their grain without duplication regardless of - query shape. +- **G2 (no duplication).** Measures aggregate over their own source rows without duplication + regardless of query shape. - **Closure.** A measure relation behaves as a relation; its field domain is independent of which measures are queried. - **Fan-out / chasm trap.** The two classic duplication errors G2 eliminates. @@ -1320,12 +1324,12 @@ joins by default matches standard SQL behavior. ### Extending the proposal -Grain manipulation is explicitly out of scope in this version (Section 5.3), so this proposal does +Evaluation controls are explicitly out of scope in this version (Section 5.3), so this proposal does not adopt the paper's `AT` operator or its consumer-modifiable *evaluation context* (a predicate over a measure's dimension columns, adjusted at the call site with `ALL`, `SET`, or `WHERE`). A -measure here is anchored to its grain (Section 3.2) and evaluated only through `MEASURE()`. +measure here is anchored to its source rows (Section 3.2) and evaluated only through `MEASURE()`. -The functionality grain manipulation enables, such as percent of total and period over period, is +The functionality evaluation controls enable, such as percent of total and period over period, is important, and we recommend strongly that it be considered as a follow-up extension. Two things are worth considering in that design. From 34feb172a6c30cd6825dbc576e964019dfb0d149 Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Tue, 13 Oct 2026 10:01:37 -0700 Subject: [PATCH 6/8] [core-spec] Rename the measure evaluation function from MEASURE() to AGG() Use AGG() as the single normative spelling throughout, so evaluating a measure is visually distinct from defining one (TO MEASURE). Engines MAY keep MEASURE() or other spellings as non-normative synonyms; portable queries and conformance tests use AGG(). Update Appendix D to explain the choice over the paper's AGGREGATE(). Co-authored-by: Isaac --- core-spec/relational_semantics.md | 212 +++++++++++++++--------------- 1 file changed, 107 insertions(+), 105 deletions(-) diff --git a/core-spec/relational_semantics.md b/core-spec/relational_semantics.md index 4dcf9f7da..34821c0f7 100644 --- a/core-spec/relational_semantics.md +++ b/core-spec/relational_semantics.md @@ -68,7 +68,7 @@ alone gives flexibility, but not correctness. This specification proposes that BI and AI tools query Ossie models through standard SQL extended with correct measure semantics ([*Measures in SQL*](https://arxiv.org/abs/2406.00251)). The engine evaluates each measure over its own source rows, and it stays correct however the consumer -filters, groups, and joins. A tool evaluates it with a single `MEASURE()` aggregate and otherwise +filters, groups, and joins. A tool evaluates it with a single `AGG()` aggregate and otherwise writes the SQL it always would. Consumers get both governed, consistent measures and the flexibility of a SQL query interface. @@ -76,28 +76,28 @@ flexibility of a SQL query interface. A consumer connects to a semantic layer, discovers a model's relations, and queries them with ordinary SQL. The one difference from querying plain tables: a measure is evaluated through the -`MEASURE()` aggregate: +`AGG()` aggregate: ```sql -- illustrative SELECT o.region, - MEASURE(o.total_revenue), - MEASURE(s.ticket_count), - MEASURE(o.total_revenue) / MEASURE(s.ticket_count) AS revenue_per_ticket + AGG(o.total_revenue), + AGG(s.ticket_count), + AGG(o.total_revenue) / AGG(s.ticket_count) AS revenue_per_ticket FROM model.orders o JOIN model.support s ON o.customer_id = s.customer_id WHERE o.order_date >= DATE '2025-01-01' GROUP BY o.region -ORDER BY MEASURE(o.total_revenue) DESC; +ORDER BY AGG(o.total_revenue) DESC; ``` `total_revenue` (on `orders`) and `ticket_count` (on `support`) are measures over different source rows, identified as measures from metadata (Section 11). `region`, `customer_id`, and `order_date` are ordinary fields. The tool joins the two tables as it would any tables and evaluates a measure -from each with `MEASURE()`. The engine computes each measure over its own source rows, so the +from each with `AGG()`. The engine computes each measure over its own source rows, so the many-to-many join on `customer_id` inflates neither one (no fan-out or chasm trap, G2), while `WHERE` and `GROUP BY` slice them like any aggregate (G1), on any field, join key or not. The same -query yields the same measure values across tools. A consuming tool writes `MEASURE(name)` where it +query yields the same measure values across tools. A consuming tool writes `AGG(name)` where it would otherwise write a raw aggregate, and generates SQL as it always has. Its obligations are in Section 8.5. @@ -139,7 +139,7 @@ It does not cover: - Physical execution, storage, indexing, or performance. The SQL spelling shown throughout is illustrative and provisional. This specification aims to -standardize the query extension: the measure column type, `MEASURE()`, and the measure-definition +standardize the query extension: the measure column type, `AGG()`, and the measure-definition marker. Its syntax is expected to be refined during working-group review rather than left for each engine to reinvent. @@ -166,7 +166,7 @@ These principles motivate the normative rules and are themselves non-normative. by name and never reconstructs its computation, so a governed metric comes out correct and identical across every consumer, however the consumer filters, groups, and joins tables. - **P3. Minimal consumer change.** Querying a measure differs from an ordinary aggregate by - exactly one substitution: a raw aggregate expression becomes `MEASURE(name)`. A tool's query + exactly one substitution: a raw aggregate expression becomes `AGG(name)`. A tool's query generation is otherwise unchanged. The placement of aggregates in SELECT, HAVING, and ORDER BY, and the use of subqueries, joins, and grouping, all carry over as written. @@ -252,7 +252,7 @@ names. The correspondence: Ossie model metadata this interface does not natively surface (`ai_context`, `custom_extensions`, and non-selected expression dialects) is addressed under discovery (Section 11) and out of scope (Section 13.2). A conforming engine MUST preserve metric semantics: a metric evaluated via -`MEASURE()` is computed from the model's metric definition (its aggregate and source rows), so two +`AGG()` is computed from the model's metric definition (its aggregate and source rows), so two consumers issuing the same query agree on its value (Section 10.2). --- @@ -273,7 +273,7 @@ central rules: predicate (which must be `boolean`, not `boolean measure`); or as a `GROUP BY` key (grouping is not defined over `T measure`). A measure column MAY still pass through a select list, which accepts a value of any type, carrying its `T measure` type onward (Section 7.1). -- **R2.** The only operation that converts `T measure` to `T` is the `MEASURE` aggregate +- **R2.** The only operation that converts `T measure` to `T` is the `AGG` aggregate function (Section 6), which is valid only in an aggregate context. A well-typed query MUST reject, at analysis time, any expression that violates R1 or R2. For @@ -295,7 +295,7 @@ SELECT total_sales FROM v GROUP BY total_sales; A value of type `T measure` is produced only by a **measure definition** (Section 5), the construct that marks an aggregate expression as a measure. That is also the one place an aggregate expression may appear outside an aggregated query; everywhere else an aggregate, -including `MEASURE()`, is valid only in an aggregated context (R2). +including `AGG()`, is valid only in an aggregated context (R2). ### 4.3 Propagation @@ -335,21 +335,21 @@ Normatively: ### 5.2 Derived measures A measure definition's expression is an ordinary SQL aggregate expression, so every value in it -comes from an aggregate function: `MEASURE()` reads an existing measure, just as `SUM` or `COUNT` -aggregates a raw field. A definition that reads one or more existing measures with `MEASURE()` +comes from an aggregate function: `AGG()` reads an existing measure, just as `SUM` or `COUNT` +aggregates a raw field. A definition that reads one or more existing measures with `AGG()` yields a derived measure: ```sql -- illustrative -avg_order_value := (MEASURE(total_revenue) / COUNT(DISTINCT order_id)) TO MEASURE +avg_order_value := (AGG(total_revenue) / COUNT(DISTINCT order_id)) TO MEASURE ``` -`MEASURE(total_revenue)` reads the existing revenue measure and `COUNT(DISTINCT order_id)` +`AGG(total_revenue)` reads the existing revenue measure and `COUNT(DISTINCT order_id)` counts orders from a raw field. A derived measure delegates to the measures it references: each is evaluated over its own source rows, and the surrounding scalar expression is applied after those -evaluations (see Section 7.2). Its inner `MEASURE()` calls are captured unevaluated, just as +evaluations (see Section 7.2). Its inner `AGG()` calls are captured unevaluated, just as `TO MEASURE` captures `SUM(x)`: they are evaluated at query time, in the consuming query's -context (not when the derived measure is defined), so no `MEASURE()` is evaluated outside an +context (not when the derived measure is defined), so no `AGG()` is evaluated outside an aggregated context (R2). Derived measures MAY reference measures from more than one relation; such a measure is a **multi-table measure** (Section 10.3). @@ -363,51 +363,51 @@ document. When present, they MUST preserve the no-duplication guarantee of Secti --- -## 6. Querying measures: the MEASURE function +## 6. Querying measures: the AGG function -A measure is evaluated through the aggregate function `MEASURE`: +A measure is evaluated through the aggregate function `AGG`: -- **Signature.** `MEASURE(m)` takes an argument of type `T measure` and returns a value of +- **Signature.** `AGG(m)` takes an argument of type `T measure` and returns a value of type `T`. -- **Kind.** `MEASURE` is an aggregate function. It MUST be treated syntactically and +- **Kind.** `AGG` is an aggregate function. It MUST be treated syntactically and semantically as an aggregate everywhere: it MAY appear wherever a standard aggregate may appear (e.g. SELECT list, HAVING, ORDER BY), and it MUST NOT appear where a standard aggregate may not. Placing it in a SELECT list or `HAVING` makes the query aggregated (Section 7.2), just as any aggregate does. - **Argument.** The argument MUST be an expression of type `T measure`. In this version, the only such expression is a reference to a measure column, optionally qualified by relation - alias (`MEASURE(o.revenue)`). Because no operator or function is defined on `T measure` (R1), - a scalar expression over measures, such as `MEASURE(a + b)`, is not well-typed. Write - `MEASURE(a) + MEASURE(b)` instead. + alias (`AGG(o.revenue)`). Because no operator or function is defined on `T measure` (R1), + a scalar expression over measures, such as `AGG(a + b)`, is not well-typed. Write + `AGG(a) + AGG(b)` instead. Later extensions may add other expressions of type `T measure`, such as a measure whose input set is modified, or an anonymous inline measure. This rule admits them without change. -`MEASURE(m)` evaluates `m`'s calculation over its input set: the source rows of `m` that remain +`AGG(m)` evaluates `m`'s calculation over its input set: the source rows of `m` that remain in each group after the enclosing query's joins and filters (Section 8.1). -> An engine MAY expose an additional spelling of this function (for example `AGG`) as a -> synonym. Synonyms are non-normative. +> An engine MAY expose additional spellings of this function (for example `MEASURE()`) as +> synonyms. Synonyms are non-normative: portable queries and conformance tests use `AGG()`. --- ## 7. Operator semantics This section specifies how **measure columns** behave under each SQL operator. The semantics -follow from the definitions of a measure column and the `MEASURE()` function: +follow from the definitions of a measure column and the `AGG()` function: - A measure column has the new type `T measure`, which SQL's operators and functions are not defined on. By R1 (Section 4.1) it cannot be an operand of a scalar operator or function, nor be used where a specific type is required, e.g. a `WHERE` or join predicate must be `boolean` (not `boolean measure`). Grouping is not defined over `T measure`. It passes through a select list unchanged, keeping its `T measure` type. -- `MEASURE()` is an aggregate function (Section 6). Ordinary SQL then fixes where it may appear: +- `AGG()` is an aggregate function (Section 6). Ordinary SQL then fixes where it may appear: in a `SELECT` list, `HAVING`, or `ORDER BY` (and inside scalar expressions there), and not in `WHERE` or `GROUP BY`, just as `SUM(...)` may not. -- Once `MEASURE()` has evaluated a measure to a field of type `T`, the result is an ordinary +- Once `AGG()` has evaluated a measure to a field of type `T`, the result is an ordinary column, with no special rules thereafter. Every operator then behaves as in standard SQL. The only measure-specific behavior is that each -`MEASURE()` is evaluated over its measure's own input set (Section 8.1). That is what makes +`AGG()` is evaluated over its measure's own input set (Section 8.1). That is what makes filtering, grouping, and joining compose correctly (G1, G2). The subsections state only that behavior and the few outright exceptions; the rest is standard SQL and is not restated. @@ -415,11 +415,11 @@ The following table summarizes; the subsections are normative. | Operator | Measure columns | | :-- | :-- | -| SELECT, unaggregated (7.1) | Pass through as `T measure`; may be renamed; `*` includes them; may be (re)defined. Evaluating one via `MEASURE()` makes the query aggregated (7.2). | -| SELECT, aggregated (7.2) | Measures allowed only inside `MEASURE()`. A bare (unevaluated) measure in an aggregated select is an error. | -| WHERE (7.3) | Ordinary predicate; filters the measure's input rows before evaluation. `MEASURE()` only via a scalar subquery. | +| SELECT, unaggregated (7.1) | Pass through as `T measure`; may be renamed; `*` includes them; may be (re)defined. Evaluating one via `AGG()` makes the query aggregated (7.2). | +| SELECT, aggregated (7.2) | Measures allowed only inside `AGG()`. A bare (unevaluated) measure in an aggregated select is an error. | +| WHERE (7.3) | Ordinary predicate; filters the measure's input rows before evaluation. `AGG()` only via a scalar subquery. | | GROUP BY / GROUPING SETS / CUBE / ROLLUP (7.4) | Grouping ranges over fields; grouping is not defined over `T measure`. `GROUP BY ALL` groups by the non-measure select expressions. | -| HAVING / ORDER BY (7.5) | `MEASURE()` allowed, as for any aggregate. | +| HAVING / ORDER BY (7.5) | `AGG()` allowed, as for any aggregate. | | JOIN (7.6) | Any join type/condition over fields. Each measure retains its own row identity and is de-duplicated before evaluation (G2). Joining on a measure is an error. | | Subquery / CTE (7.7) | Measures propagate as `T measure` with row identity intact. | | Set operators, DISTINCT (7.8) | Error when an input carries an unevaluated measure. | @@ -432,9 +432,9 @@ In a SELECT without aggregation, a measure column is treated like any other colu type and its row identity; if not projected, it is dropped; - it MAY be defined or derived by a measure definition in the projection (Section 5); - it MUST NOT be the operand of a scalar operator or function (R1); -- it is not evaluated to a value here: `MEASURE()` is the only way to evaluate a measure, and a - `MEASURE()` in the select list makes the query aggregated (Section 7.2), so a genuinely - unaggregated projection contains no `MEASURE()` (R2). +- it is not evaluated to a value here: `AGG()` is the only way to evaluate a measure, and a + `AGG()` in the select list makes the query aggregated (Section 7.2), so a genuinely + unaggregated projection contains no `AGG()` (R2). Non-normative examples: @@ -449,21 +449,21 @@ SELECT total_sales + 1 FROM v; -- rejected: scalar op on a me A SELECT is aggregated (as in standard SQL) if it has a `GROUP BY` clause, `HAVING` clause, or an aggregate expression in its select list. In an aggregated SELECT: -- A measure column MUST appear only inside `MEASURE()`. A bare, unevaluated measure in the +- A measure column MUST appear only inside `AGG()`. A bare, unevaluated measure in the select list of an aggregated query MUST be rejected. -- `MEASURE(m)` returns a scalar of type `T`; it MAY be combined with other scalars and +- `AGG(m)` returns a scalar of type `T`; it MAY be combined with other scalars and aggregates in the select list, HAVING, and ORDER BY. -- For a derived measure or an expression combining several `MEASURE()` calls, each `MEASURE()` is +- For a derived measure or an expression combining several `AGG()` calls, each `AGG()` is evaluated independently over its own input set and the surrounding scalar expression applied per - output group (Section 8.1). For example, `MEASURE(revenue) / MEASURE(ticket_count)` divides two + output group (Section 8.1). For example, `AGG(revenue) / AGG(ticket_count)` divides two independently duplication-free aggregates within each group. Non-normative examples: ```sql -SELECT region, MEASURE(total_revenue) FROM v GROUP BY region; -- ok -SELECT MEASURE(total_sales) FROM v; -- ok: aggregated grand total -SELECT region, MAX(order_date), MEASURE(total_revenue) FROM v +SELECT region, AGG(total_revenue) FROM v GROUP BY region; -- ok +SELECT AGG(total_sales) FROM v; -- ok: aggregated grand total +SELECT region, MAX(order_date), AGG(total_revenue) FROM v GROUP BY region; -- ok: an ordinary aggregate over a field, beside a measure SELECT region, total_revenue FROM v GROUP BY region; -- rejected: bare measure (R1) ``` @@ -472,18 +472,18 @@ SELECT region, total_revenue FROM v GROUP BY region; -- rejected: bare measure `WHERE` filters the input rows before aggregation, so a measure sees only the surviving rows: filtering on a field changes a measure just as it would a raw aggregate over the same rows -(half of Guarantee G1, Section 8.2). A scalar subquery in the predicate MAY use `MEASURE()`; a +(half of Guarantee G1, Section 8.2). A scalar subquery in the predicate MAY use `AGG()`; a predicate on a measure's value goes in `HAVING`. Non-normative examples: ```sql -SELECT region, MEASURE(total_revenue) FROM v +SELECT region, AGG(total_revenue) FROM v WHERE order_date >= DATE '2025-01-01' GROUP BY region; -- ok: WHERE filters the measure's input -SELECT region, MEASURE(total_revenue) FROM v -WHERE MEASURE(total_revenue) > 100 GROUP BY region; -- rejected: aggregate in WHERE; use HAVING -SELECT region, MEASURE(total_revenue) FROM v -WHERE (SELECT MEASURE(gross) FROM budget) > 100 GROUP BY region; -- ok: MEASURE() in a subquery +SELECT region, AGG(total_revenue) FROM v +WHERE AGG(total_revenue) > 100 GROUP BY region; -- rejected: aggregate in WHERE; use HAVING +SELECT region, AGG(total_revenue) FROM v +WHERE (SELECT AGG(gross) FROM budget) > 100 GROUP BY region; -- ok: AGG() in a subquery ``` ### 7.4 GROUP BY @@ -498,21 +498,21 @@ WHERE (SELECT MEASURE(gross) FROM budget) > 100 GROUP BY region; -- ok: MEASURE Non-normative examples: ```sql -SELECT region, MEASURE(total_revenue) FROM v GROUP BY region; -- ok -SELECT region, MEASURE(total_revenue) FROM v GROUP BY ALL; -- ok, groups by region +SELECT region, AGG(total_revenue) FROM v GROUP BY region; -- ok +SELECT region, AGG(total_revenue) FROM v GROUP BY ALL; -- ok, groups by region SELECT region FROM v GROUP BY total_revenue; -- rejected: grouping not defined over a measure (R1) ``` ### 7.5 HAVING and ORDER BY -Since `MEASURE()` is an aggregate (Section 6), `HAVING` and `ORDER BY` MAY contain `MEASURE()`, +Since `AGG()` is an aggregate (Section 6), `HAVING` and `ORDER BY` MAY contain `AGG()`, and scalar expressions over it, just as they may `SUM(...)`. Non-normative example: ```sql -SELECT region, MEASURE(total_revenue) FROM v GROUP BY region -HAVING MEASURE(total_revenue) > 100 ORDER BY MEASURE(total_revenue) DESC; -- ok +SELECT region, AGG(total_revenue) FROM v GROUP BY region +HAVING AGG(total_revenue) > 100 ORDER BY AGG(total_revenue) DESC; -- ok ``` ### 7.6 JOIN @@ -541,7 +541,7 @@ Non-normative example (many-to-many join on `region`, each measure still duplica SELECT s.region, t.ticket_count, s.total_sales, - MEASURE(s.total_sales) / MEASURE(t.ticket_count) TO MEASURE AS sales_per_ticket + AGG(s.total_sales) / AGG(t.ticket_count) TO MEASURE AS sales_per_ticket FROM tickets t JOIN sales s ON t.region = s.region; ``` @@ -558,7 +558,7 @@ by this specification (Section 13.1); intermediate propagation is fully specifie Non-normative example: ```sql -SELECT region, MEASURE(total_revenue) +SELECT region, AGG(total_revenue) FROM (SELECT region, total_revenue FROM v WHERE order_date >= DATE '2025-01-01') t GROUP BY region; -- total_revenue stays a measure in the subquery, evaluated at the top ``` @@ -571,7 +571,7 @@ unevaluated measure has no value to compare or append. Therefore: - A set operator or `DISTINCT` applied to an input relation that carries one or more unevaluated measure columns MUST be rejected. - The same operators are valid once measures have been evaluated to scalars (via - `MEASURE()` in an aggregated subquery) or projected away. For example, a `UNION ALL` of + `AGG()` in an aggregated subquery) or projected away. For example, a `UNION ALL` of two subqueries that each select only fields is valid. Non-normative examples: @@ -592,7 +592,7 @@ query. The consumer-side obligations these guarantees assume are collected in Se ### 8.1 Evaluation semantics -A measure is evaluated only through `MEASURE()` in an aggregated query (Sections 6, 7.2). Its +A measure is evaluated only through `AGG()` in an aggregated query (Sections 6, 7.2). Its value in each output group is defined by the set of source rows it aggregates in that group. **Source rows.** A measure `m`'s **source rows** are the rows produced by the `FROM` clause in @@ -617,7 +617,7 @@ These rules apply at every query level between `m`'s definition and its evaluati the defining query and the evaluating query. Derivation is tracked separately for each measure column, so in a self-join, each side's measure follows its own side's rows. -**Input set.** In an aggregated query, the **input set** of `MEASURE(m)` for an output group `g` +**Input set.** In an aggregated query, the **input set** of `AGG(m)` for an output group `g` is the set of `m`'s source rows from which at least one row of `g` derives. `m`'s aggregate is evaluated over the input set, each source row contributing once, with its own argument values. If the input set is empty (filters removed every source row, or no row of `g` derives from a @@ -640,7 +640,7 @@ vanishing from the result. Each measure in a query is evaluated this way independently, over its own input set. A derived measure (Section 5.2) evaluates each measure it references over that measure's own input set -for the same group. An expression combining several `MEASURE()` calls applies its scalar +for the same group. An expression combining several `AGG()` calls applies its scalar operators to their per-group results, following the host dialect's standard scalar and NULL semantics, including its own handling of cases such as division by zero (Section 7.2). @@ -649,7 +649,7 @@ same results. ### 8.2 G1: Slice-and-dice responsiveness -`MEASURE(m)` responds to `WHERE` and `GROUP BY` just as a standard SQL aggregate over `m`'s +`AGG(m)` responds to `WHERE` and `GROUP BY` just as a standard SQL aggregate over `m`'s definition would: a `WHERE` predicate restricts the input rows `m` sees (Section 7.3), and the query's grouping sets the level at which `m` is reported (Section 7.4). This follows directly from Section 8.1: filters remove source rows from `m`'s input set, and each group's input @@ -659,11 +659,11 @@ aggregates. ### 8.3 G2: No duplication -`MEASURE(m)` aggregates `m` over its own source rows without duplication, regardless of the shape of +`AGG(m)` aggregates `m` over its own source rows without duplication, regardless of the shape of the enclosing query: any join type and cardinality, and any other measures or relations present. This follows from Section 8.1: `m`'s input set is a set of source rows, so each source row contributes to `m`'s aggregate at most once per output group, however many times a join duplicated -it, and null-padded rows from an outer join do not contribute at all. `MEASURE(m)` therefore does +it, and null-padded rows from an outer join do not contribute at all. `AGG(m)` therefore does not depend on the cardinality of any other relation or measure in the query. G2 structurally eliminates the two classic multi-table aggregation errors: @@ -677,20 +677,20 @@ G2 structurally eliminates the two classic multi-table aggregation errors: ### 8.4 Corollary: Closure and stable field domains -Because `MEASURE()` behaves as an ordinary SQL aggregate (G1), a measure relation behaves as an +Because `AGG()` behaves as an ordinary SQL aggregate (G1), a measure relation behaves as an ordinary relation: the set of groups a query returns (its field domain) is fixed by its grouping fields and filters, just as with standard `GROUP BY`, and does not depend on which measures it selects. In particular, for a measure relation `R`, the set of field values returned by ```sql -SELECT d, MEASURE(m1) FROM R GROUP BY d +SELECT d, AGG(m1) FROM R GROUP BY d ``` and by ```sql -SELECT d, MEASURE(m2) FROM R GROUP BY d +SELECT d, AGG(m2) FROM R GROUP BY d ``` MUST be the same (the domain of `d`), differing only in the measure column. A conforming @@ -709,14 +709,14 @@ engine MUST preserve this closure property. The guarantees above hold when a consuming tool observes the following. Unlike G1-G2, these are requirements on the *consumer*, not the engine. -- **Evaluate measures only through `MEASURE()`.** A consumer references a measure by name and lets +- **Evaluate measures only through `AGG()`.** A consumer references a measure by name and lets the engine evaluate it (P2); it MUST NOT reconstruct a measure from the underlying columns, which would forgo the no-duplication guarantee. -- **Do not reaggregate a `MEASURE()` result on the client.** A `MEASURE()` value is already +- **Do not reaggregate an `AGG()` result on the client.** An `AGG()` value is already aggregated at the query's grouping; combining such values across groups on the client (for a coarser grouping, a drill-up, or a cached-then-refiltered view) is not valid, the same rule that applies to non-decomposable aggregates such as `COUNT(DISTINCT ...)` or `MEDIAN`. To - change the grouping, re-query the engine. For example, given per-state `MEASURE(revenue)`, a + change the grouping, re-query the engine. For example, given per-state `AGG(revenue)`, a tool MUST NOT sum those client-side for a national total; it re-queries grouped at the national level. @@ -783,10 +783,10 @@ A multi-table Ossie model is mapped to a set of relations (database tables, view relations), each with its own fields and measures, together with a set of relationships among them. A tool discovers the relations and relationships (Section 11), then -generates queries that join them using its own SQL, reading measures via `MEASURE()`. +generates queries that join them using its own SQL, reading measures via `AGG()`. Nothing in a tool's multi-table workflow needs to change: the tool joins measure relations -just as it joins any table, and the only substitution is `MEASURE(name)` in place of a raw +just as it joins any table, and the only substitution is `AGG(name)` in place of a raw aggregate. The engine supplies G1 and G2 across the joins. Independent measure relations (multi-fact models) are supported directly: a query MAY join @@ -841,14 +841,14 @@ join is defined on a relation whose own definition includes that join, so its ro is well-defined at the point of definition. For example, a measure `SUM(qty * price)` reading columns from both `orders` and `products` is a measure on a relation defined `FROM orders JOIN products`, evaluated at that joined relation's row identity; the consumer -evaluates it with `MEASURE()` like any other. This is the general pattern: fold a measure's +evaluates it with `AGG()` like any other. This is the general pattern: fold a measure's required joins into its home relation's definition, rather than leaving them to the consuming query. Multi-relation measures fit the same pattern. A measure derived from measures on more than one relation (say a ratio of a measure from each of two facts, Section 5.2) lives on a home relation whose field domain is fixed, such as a conformed dimension shared by the two facts. -Defined there as `(MEASURE(a) / MEASURE(b)) TO MEASURE`, it is an ordinary measure relation that +Defined there as `(AGG(a) / AGG(b)) TO MEASURE`, it is an ordinary measure relation that obeys closure (Section 8.4); Appendix A.7 works this through with a conformed-dimension bridge. A free-floating measure (attached to no relation, evaluated over whatever join path each query @@ -1087,14 +1087,14 @@ Deferred interface and interop concerns: ## Appendix A. Worked examples (non-normative) -These illustrate that ordinary tool-generated SQL, using only the `MEASURE()` substitution, +These illustrate that ordinary tool-generated SQL, using only the `AGG()` substitution, composes with the guarantees above. They assume a model with measure relations `orders` (one row per order), `sales` (one row per sale), and `tickets` (one row per ticket). ### A.1 Simple aggregation ```sql -SELECT category, MEASURE(total_quantity) +SELECT category, AGG(total_quantity) FROM orders GROUP BY category; ``` @@ -1106,10 +1106,10 @@ GROUP BY category; A tool keeps the top 2 states by revenue by computing them in a subquery and joining back: ```sql -SELECT o.category, MEASURE(o.total_quantity) +SELECT o.category, AGG(o.total_quantity) FROM orders o JOIN ( - SELECT state_id, MEASURE(total_revenue) AS r + SELECT state_id, AGG(total_revenue) AS r FROM orders GROUP BY state_id ORDER BY r DESC, state_id @@ -1118,7 +1118,7 @@ JOIN ( GROUP BY o.category; ``` -The join filters rows before `MEASURE(o.total_quantity)` is evaluated (G1) and does not inflate +The join filters rows before `AGG(o.total_quantity)` is evaluated (G1) and does not inflate it (G2). Because the interface is standard relational SQL, the `state_id` values the subquery ranks are exactly the `state_id` values it joins back on. The field domain is fixed by the relation and its filters, independent of which measure is selected (closure, Section 8.4). A @@ -1134,22 +1134,22 @@ Average, over states, of each state's total revenue: ```sql SELECT d.category, AVG(m.state_revenue) FROM (SELECT category, state_id FROM orders GROUP BY category, state_id) d -JOIN (SELECT state_id, MEASURE(total_revenue) AS state_revenue +JOIN (SELECT state_id, AGG(total_revenue) AS state_revenue FROM orders GROUP BY state_id) m ON d.state_id IS NOT DISTINCT FROM m.state_id GROUP BY d.category; ``` -The inner `MEASURE(total_revenue)` is duplication-free per state (G2); the outer `AVG` is an +The inner `AGG(total_revenue)` is duplication-free per state (G2); the outer `AVG` is an ordinary aggregate over the already-evaluated scalar. ### A.4 Multi-fact ratio ```sql SELECT s.region, - MEASURE(t.ticket_count), - MEASURE(s.total_sales), - MEASURE(s.total_sales) / MEASURE(t.ticket_count) AS sales_per_ticket + AGG(t.ticket_count), + AGG(s.total_sales), + AGG(s.total_sales) / AGG(t.ticket_count) AS sales_per_ticket FROM tickets t JOIN sales s ON t.region = s.region GROUP BY s.region; ``` @@ -1162,13 +1162,13 @@ evaluated over its own source rows (G2), and the ratio is applied per region aft Revenue per category, with the fact joined to its line items (one order -> many items): ```sql -SELECT o.category, MEASURE(o.total_revenue) +SELECT o.category, AGG(o.total_revenue) FROM orders o JOIN order_items i ON i.order_id = o.order_id GROUP BY o.category; ``` -The join produces one row per order item, but `MEASURE(o.total_revenue)` aggregates each order +The join produces one row per order item, but `AGG(o.total_revenue)` aggregates each order once (G2): each order's revenue is counted once, not once per line item. ### A.6 Join type: the tool's choice @@ -1177,21 +1177,21 @@ The same measures under an inner and a left join, differing only in the join typ ```sql -- inner: only regions with both sales and tickets -SELECT s.region, MEASURE(s.total_sales), MEASURE(t.ticket_count) +SELECT s.region, AGG(s.total_sales), AGG(t.ticket_count) FROM sales s JOIN tickets t ON s.region = t.region GROUP BY s.region; -- left: every sales region; ticket_count is 0 where a region has no tickets -SELECT s.region, MEASURE(s.total_sales), MEASURE(t.ticket_count) +SELECT s.region, AGG(s.total_sales), AGG(t.ticket_count) FROM sales s LEFT JOIN tickets t ON s.region = t.region GROUP BY s.region; ``` The join type is the tool's analytic choice, not something the interface prescribes (Section 10.2). Either way each measure is evaluated over its own source rows (G2). The left join keeps a region -that has sales but no tickets; there `MEASURE(t.ticket_count)` has no source rows and takes its +that has sales but no tickets; there `AGG(t.ticket_count)` has no source rows and takes its empty-input value (0 for `COUNT`, Section 8.1), while the inner join omits that region entirely. For -any region in both results the `MEASURE()` values are identical: the standard fixes measure values +any region in both results the `AGG()` values are identical: the standard fixes measure values for a given query, and the join choice fixes which rows appear. ### A.7 Multi-fact via a conformed-dimension bridge @@ -1207,7 +1207,7 @@ view: CREATE VIEW region_metrics AS SELECT r.region_id, -- field r.region_name, -- field - (MEASURE(sales.total_sales) / MEASURE(tickets.ticket_count)) + (AGG(sales.total_sales) / AGG(tickets.ticket_count)) TO MEASURE AS sales_per_ticket -- derived measure FROM regions r LEFT JOIN sales ON sales.region_id = r.region_id @@ -1215,7 +1215,7 @@ LEFT JOIN tickets ON tickets.region_id = r.region_id; ``` `region_metrics` carrying `sales_per_ticket` is then an ordinary measure relation. A consumer -evaluates `MEASURE(sales_per_ticket)` and MAY join `region_metrics` to further tables with +evaluates `AGG(sales_per_ticket)` and MAY join `region_metrics` to further tables with ordinary SQL: the measure stays duplication-free (G2), its `region_id` domain is fixed (closure, Section 8.4), and no model re-declaration is needed to bring in another table. @@ -1260,10 +1260,10 @@ Interface terms: same source row. The **row-identity key** (Section 9) is one implementation of it. - **Input set.** For a measure and an output group, the set of the measure's source rows that the group's rows derive from; the measure's aggregate is evaluated over it (Section 8.1). -- **`MEASURE(m)`.** The aggregate function that evaluates measure `m` over its input set, +- **`AGG(m)`.** The aggregate function that evaluates measure `m` over its input set, returning `T`. - **Base / derived measure.** A measure defined directly from an aggregate expression, versus - one defined from `MEASURE()` over other measures. + one defined from `AGG()` over other measures. - **Multi-table measure.** A derived measure referencing measures from more than one relation. - **G1 (slice-and-dice).** Measures respond to WHERE and GROUP BY like standard aggregates. @@ -1298,10 +1298,12 @@ the proposal later. ### Naming and syntax -The evaluation function is `MEASURE()`, not the paper's `AGGREGATE()`. `AGGREGATE` already names a -different function in some engines (in Databricks it is a higher-order function over arrays), and -the evaluation function needs a name unlikely to collide with existing functions or reserved words -across popular SQL engines. +The evaluation function is `AGG()`, not the paper's `AGGREGATE()`. `AGGREGATE` already names a +different function in some engines (in Databricks and DuckDB it is a function over arrays or +lists), and the evaluation function needs a name unlikely to collide with existing functions or +reserved words across popular SQL engines. Earlier drafts of this specification used `MEASURE()`, +which reads too much like the `TO MEASURE` definition marker; engines MAY keep it as a synonym +(Section 6). A measure is defined with `expr TO MEASURE AS name`, not the paper's `expr AS MEASURE name`. `AS MEASURE` is cleaner, but ambiguous: in standard SQL `expr AS measure` already means "give this @@ -1309,15 +1311,15 @@ column the alias `measure`." ### Evaluating a measure -A measure is evaluated only through `MEASURE()` (R2). The paper treats its `AGGREGATE()` wrapper +A measure is evaluated only through `AGG()` (R2). The paper treats its `AGGREGATE()` wrapper as largely cosmetic: in an aggregate query a bare measure reference is evaluated implicitly. We require the explicit call, so the conversion of a `T measure` to a scalar `T` happens at one visible place. Further, in SQL, an aggregate function in the `SELECT`, `HAVING`, or `ORDER BY` clause is what turns a projection into an aggregation; a bare measure column that implicitly -aggregates breaks that expectation, so requiring `MEASURE()` keeps the query syntactically +aggregates breaks that expectation, so requiring `AGG()` keeps the query syntactically ordinary SQL. -`MEASURE()` evaluates a measure over exactly the rows the query selects, after its `WHERE` filters +`AGG()` evaluates a measure over exactly the rows the query selects, after its `WHERE` filters and joins (Guarantee G1). In contrast, the paper defaults to not applying the query's `WHERE` or joins to a measure; a consumer opts those in with `VISIBLE`. Applying the query's filters and joins by default matches standard SQL behavior. @@ -1327,7 +1329,7 @@ joins by default matches standard SQL behavior. Evaluation controls are explicitly out of scope in this version (Section 5.3), so this proposal does not adopt the paper's `AT` operator or its consumer-modifiable *evaluation context* (a predicate over a measure's dimension columns, adjusted at the call site with `ALL`, `SET`, or `WHERE`). A -measure here is anchored to its source rows (Section 3.2) and evaluated only through `MEASURE()`. +measure here is anchored to its source rows (Section 3.2) and evaluated only through `AGG()`. The functionality evaluation controls enable, such as percent of total and period over period, is important, and we recommend strongly that it be considered as a follow-up extension. Two things From f911d3b74ffcd290d05a2795e45e869ab140436c Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Tue, 13 Oct 2026 10:57:14 -0700 Subject: [PATCH 7/8] [core-spec] State measure placement rules for measure-typed expressions Phrase the aggregated-SELECT rule and the Section 7 type summary in terms of expressions of type T measure rather than measure columns, matching the generalized AGG() argument rule. Rules that are genuinely about columns (relations, propagation, discovery) are unchanged. Co-authored-by: Isaac --- core-spec/relational_semantics.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/core-spec/relational_semantics.md b/core-spec/relational_semantics.md index 34821c0f7..9b174a76c 100644 --- a/core-spec/relational_semantics.md +++ b/core-spec/relational_semantics.md @@ -139,7 +139,7 @@ It does not cover: - Physical execution, storage, indexing, or performance. The SQL spelling shown throughout is illustrative and provisional. This specification aims to -standardize the query extension: the measure column type, `AGG()`, and the measure-definition +standardize the query extension: the measure type, `AGG()`, and the measure-definition marker. Its syntax is expected to be refined during working-group review rather than left for each engine to reinvent. @@ -396,10 +396,11 @@ This section specifies how **measure columns** behave under each SQL operator. T follow from the definitions of a measure column and the `AGG()` function: - A measure column has the new type `T measure`, which SQL's operators and functions are not - defined on. By R1 (Section 4.1) it cannot be an operand of a scalar operator or function, nor - be used where a specific type is required, e.g. a `WHERE` or join predicate must be `boolean` (not - `boolean measure`). Grouping is not defined over `T measure`. It passes - through a select list unchanged, keeping its `T measure` type. + defined on. By R1 (Section 4.1), no expression of that type can be an operand of a scalar + operator or function, nor be used where a specific type is required, e.g. a `WHERE` or join + predicate must be `boolean` (not `boolean measure`). Grouping is not defined over + `T measure`. A measure column passes through a select list unchanged, keeping its + `T measure` type. - `AGG()` is an aggregate function (Section 6). Ordinary SQL then fixes where it may appear: in a `SELECT` list, `HAVING`, or `ORDER BY` (and inside scalar expressions there), and not in `WHERE` or `GROUP BY`, just as `SUM(...)` may not. @@ -449,8 +450,8 @@ SELECT total_sales + 1 FROM v; -- rejected: scalar op on a me A SELECT is aggregated (as in standard SQL) if it has a `GROUP BY` clause, `HAVING` clause, or an aggregate expression in its select list. In an aggregated SELECT: -- A measure column MUST appear only inside `AGG()`. A bare, unevaluated measure in the - select list of an aggregated query MUST be rejected. +- An expression of type `T measure` MUST appear only as the argument of `AGG()`. A bare, + unevaluated measure in the select list of an aggregated query MUST be rejected. - `AGG(m)` returns a scalar of type `T`; it MAY be combined with other scalars and aggregates in the select list, HAVING, and ORDER BY. - For a derived measure or an expression combining several `AGG()` calls, each `AGG()` is From ea6e4f8d0e9650ce16948ee9b8c5c43307287b1e Mon Sep 17 00:00:00 2001 From: Justin Talbot Date: Tue, 13 Oct 2026 11:47:19 -0700 Subject: [PATCH 8/8] [core-spec] Revise Appendix D to match the Measures in SQL paper AGG() corresponds to the paper's AGGREGATE() (evaluation AT VISIBLE); the difference is that bare measure references are rejected. Add a section on choices this specification makes where the paper is open (intermediate filters and joins, joins in the evaluating query, outer joins). Rewrite the extension considerations: where controls are written and how they compose, which filters a control may lift, and identifying dimensions. Co-authored-by: Isaac --- core-spec/relational_semantics.md | 74 +++++++++++++++++-------------- 1 file changed, 41 insertions(+), 33 deletions(-) diff --git a/core-spec/relational_semantics.md b/core-spec/relational_semantics.md index 9b174a76c..9ed19039b 100644 --- a/core-spec/relational_semantics.md +++ b/core-spec/relational_semantics.md @@ -1294,8 +1294,8 @@ This interface builds on the measure model of Hyde and Fremlin, [*Measures in SQL*](https://arxiv.org/abs/2406.00251). It keeps that model's core: the `T measure` column type, the transparent expansion of a measure into ordinary SQL, and closure (a relation with measures behaves like any other relation). The rest of this appendix records where it -differs: naming and syntax, how a measure is evaluated, and some considerations for extending -the proposal later. +differs (naming, syntax, and how a measure is evaluated), where it makes choices the paper leaves +open, and some considerations for extending the proposal later. ### Naming and syntax @@ -1312,38 +1312,46 @@ column the alias `measure`." ### Evaluating a measure -A measure is evaluated only through `AGG()` (R2). The paper treats its `AGGREGATE()` wrapper -as largely cosmetic: in an aggregate query a bare measure reference is evaluated implicitly. We -require the explicit call, so the conversion of a `T measure` to a scalar `T` happens at one -visible place. Further, in SQL, an aggregate function in the `SELECT`, `HAVING`, or `ORDER BY` -clause is what turns a projection into an aggregation; a bare measure column that implicitly -aggregates breaks that expectation, so requiring `AGG()` keeps the query syntactically -ordinary SQL. +A measure is evaluated only through `AGG()` (R2), which corresponds to the paper's `AGGREGATE()`. +The paper defines `AGGREGATE(m)` as `m` evaluated `AT (VISIBLE)`: over the rows that satisfy the +query's grouping, `WHERE` filters, and join conditions. `AGG()` behaves the same way (Section 8.1). -`AGG()` evaluates a measure over exactly the rows the query selects, after its `WHERE` filters -and joins (Guarantee G1). In contrast, the paper defaults to not applying the query's `WHERE` or -joins to a measure; a consumer opts those in with `VISIBLE`. Applying the query's filters and -joins by default matches standard SQL behavior. +The paper also lets a bare measure reference be evaluated, in a default context that ignores the +query's `WHERE` clause. This specification rejects a bare measure in an aggregated query (Section +7.2), for two reasons. First, in SQL an aggregate function is what turns a projection into an +aggregation, and implicit aggregation breaks that expectation. Second, requiring `AGG()` keeps the +conversion from `T measure` to `T` in one visible place. Evaluating a measure while ignoring the +query's filters is, in this specification's terms, an evaluation control (Section 5.3). + +### Where this specification makes choices the paper leaves open + +The paper defines evaluation informally and calls for a formal semantics as future work. Section +8.1 defines a measure's input set for every operator between its definition and its evaluation. +That settles cases the paper leaves open, in a way we believe matches its intent: + +- **Filters and joins in intermediate queries.** A `WHERE` filter or join in a subquery or view + that passes a measure through restricts the measure, just as one in the evaluating query does. +- **Joins in the evaluating query.** A measure's input set in each group is defined even when the + group keys come from another relation in the join. +- **Outer joins.** Null-padded rows contribute nothing to a measure on the padded side. ### Extending the proposal -Evaluation controls are explicitly out of scope in this version (Section 5.3), so this proposal does -not adopt the paper's `AT` operator or its consumer-modifiable *evaluation context* (a predicate -over a measure's dimension columns, adjusted at the call site with `ALL`, `SET`, or `WHERE`). A -measure here is anchored to its source rows (Section 3.2) and evaluated only through `AGG()`. - -The functionality evaluation controls enable, such as percent of total and period over period, is -important, and we recommend strongly that it be considered as a follow-up extension. Two things -are worth considering in that design. - -**Encapsulation.** The paper encapsulates a measure's *formula*: a consumer uses a measure without -seeing its definition or the tables it reads (Principle P2). We believe encapsulation should cover a -measure's *meaning* as well: a measure has a single correct result for a given query, and a -consumer can slice, dice, and combine measures but cannot reach in and change what a measure -means. An extension should keep this property: a consumer evaluates a measure over modified -input sets and combines the results, without escaping its definition. - -**Column identity.** The paper's `SET` and `ALL` modifiers identify filters by dimension, which -requires deciding when two filters are on the same dimension. SQL has no general way to decide -that two columns are the same, so a context modifier for this interface will need its own answer -to that question. +This version does not include evaluation controls (Section 5.3), and so does not adopt the paper's +`AT` operator. The functionality they enable, such as percent of total and period over period, is +important, and we strongly recommend it as a follow-up extension. An extension will need to +settle: + +- **Where controls are written.** The paper applies `AT` in the consuming query. Controls could + also be declared in a measure's definition, or in any query between the definition and the + evaluation. An extension may allow any of these. Where controls apply at more than one level, + they should compose so that a level closer to the consumer cannot override what a level closer + to the definition specified. The paper's composition rule, under which outer modifiers apply + first and inner ones last, has this property. +- **Which filters a control may lift.** The paper's `SET` can reach rows that the evaluating + query's `WHERE` removed. An extension must say which filters can be lifted this way. Filters in + intermediate queries, such as a view that restricts rows for security, should not be. +- **Identifying dimensions.** `SET` and `ALL` act on the terms for a given dimension. The paper + leaves open how terms on related expressions are matched (for example `YEAR(date)` and `date`) + and how dimensions are linked. An extension needs an answer that works over arbitrary SQL + expressions.