Skip to content

feat(profile): whole-product profiles with default statistics, and FLUID SQL/1 date moves - #1024

Merged
fas89 merged 6 commits into
mainfrom
claude/profile-sql-dates-6decaf
Nov 19, 2026
Merged

fas89 merged 6 commits into
mainfrom
claude/profile-sql-dates-6decaf

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What this does

fluid profile can profile a whole data product in one call, and without --bundle it now publishes the statistics fluxsim init reads, declared from the contract alone and checked like a Simulation's fidelity block. FLUID SQL/1 gains whole-unit date moves. Six commits, each green on its own:

  1. fix(profile): a repeated --expose is refused (profile_expose_repeated) instead of profiling one expose's records as another's.
  2. docs(rfc): RFC-data-rules.md decision 14 is amended. FLUID SQL/1 gains ADD_DAYS, ADD_MONTHS and ADD_YEARS with flux-expr/3's meaning: the day clamps to the end of a shorter month, NULL in gives NULL out, and a result outside 0001-01-01..9999-12-31 is NULL. It adds a per-dialect rendering table and a note that sqlglot is never used to transpile ADD_MONTHS. Section 1.4 is corrected: fluid profile does not measure invariants yet.
  3. feat(fluid-sql): the lexer, parser, checker, exact evaluator, canonical text, the six dialect renderers (each on a count a CASE has made NULL where the result would leave the range) and the CEL refusal. Conformance vectors are 1.1.0, with 51 new vectors. fluid-schema-0.7.7.json is byte-identical.
  4. feat(profile): --expose repeats, each paired with --source <exposeId>=<location> (and --table <exposeId>=<table>). One call writes one document holding every expose. One --expose with one bare --source works as before.
  5. fix(profile): a CSV date column reads its YYYY-MM-DD text as that day.
  6. feat(profile): default statistics without --bundle (below).

Children per parent for each semantics.relationships entry moves to a follow-up PR.

Default statistics

The profiler writes its own fidelity block from the contract and answers it through the --bundle path, so flux-stats' grammar parses it. Every statistic goes through check_edge_counts (ADR 0012) and the ACRO rules. No bin edge ever comes from the data.

column statistic
enum(...) levelHistogram/1 over every declared level; a level no record holds is a withheld cell (C)
number whose validationRules close a range inside its type marginalHistogram/1, about ten bins of one width; each value is checked exactly against the range
any other bounded number, every date and timestamp quantile/2 at 0.25, 0.5 and 0.75 on a ladder fixed by the type (1-2-5 steps within the type; one bin per year 1900 to 2100)
enum with a categorical semantics.dimensions entry the other enums' level histograms per declared level, at most 10,000 cells per column

Choices the disclosure audit forced:

  • a column gets a histogram or quantiles, never both: checked together, they let a reader who knows the check's choices tell worlds apart;
  • no histogram on a type's ladder: its empty end bins publish nothing;
  • a number with a range histogram withholds sum, min, max, quantiles and distinctCount (D) and forms no mean/1, since each gave a withheld bin back;
  • a one-sided range, or one with a side at the type's own bound, takes the ladder: ten bins over the rest of the type would hold every record in one bin;
  • grouped statistics are level histograms only, capped per column because checking them costs about the cube of the levels.

A record outside what the contract declares refuses the profile (profile_record_outside_declared_domain, naming the column and row only). This is deliberate: dropping just that column's statistic would make the document's shape depend on the records.

A profile holding quantile/2 names format 1-1-0, so most profiles without --bundle are 1-1-0 documents now. flux-stats reads them from 0.7, and the [profile] pin stays flux-stats>=0.7,<0.8.

Disclosure audit

tests/profiler/test_disclosure_oracle.py enumerates every world (a multiset of records) of tiny instances: 1 or 2 tables, at most 3 bins, at most 7 rows. It runs the real profiler on each world, groups the worlds by the canonical bytes they publish (so the publish decisions are part of what a reader sees), and fails when a protected count (below the minimum cell, or a zero) is the same in every world of a group but was not already fixed without the new statistics.

Larger runs (not in the suite):

table worlds rules pins
level histogram 17,296 ACRO defaults 0
grouped level histogram 46,376 ACRO defaults 0
grouped, nulls in both columns, up to 8 rows 24,310 each of the 4 scaled audit rules 0
range histogram 12,341 ACRO defaults 0
integer on its ladder 23,751 ACRO defaults 0
date on its ladder 23,751 ACRO defaults 0

The audit and the reviews found four channels, all closed here:

  • a column's distinctCount says how many withheld cells are empty, so a column a level histogram counts, or a grouping groups by, withholds it (D), with or without --bundle;
  • a number's exact sum, min, max, quantiles and distinctCount gave a withheld bin of a range back (224 pins in 12,341 worlds before the fix);
  • a histogram and quantiles on the same bins, checked together;
  • a grouped statistic's sampleSize banded the records its groups hold. Beside the columns' own counts, that fixed the number of records with a value and a null key. It now bands the column's non-null records, on --bundle too.

Reviews

  • Security and disclosure review: BLOCK on the first pass (the grouped sampleSize band). After the fix the reviewer re-ran its own audit (24,310 worlds, 0 pins) and gave SHIP.
  • Re-check before merge: commits 1 to 6 clear. Children per parent needs a further disclosure fix and moves to its own PR. This PR also masks a database URI after a mistyped exposeId in every refusal.
  • Delivery review: CONCERNS, two items. One-sided ranges are fixed. The out-of-domain refusal is kept, for the reason above.
  • Defect scan (three hunters, a skeptic batch, a referee): 9 findings in these six commits, none refuted, all fixed, 6 of them with a regression test (the two in children per parent move with it). They were:
    • a non-list validationRules (it crashed);
    • a value just past a range bound on a coarse grid;
    • the integer top at --fractional-bits 0;
    • one-sided ranges;
    • the refusal hint not naming the column type;
    • the cost of large groupings;
    • the stand-in registry skipping the real reader for 1-0-0 documents;
    • the docs on format 1-1-0 and the help text for the default path.

Each new regression test fails on the pre-fix head and passes here.

Validation

  • Every commit on its own: ruff, the dead-code gate, black 24.10.0, scripts/mypy_strict.py (the strict allowlist now includes fluid_build.profiler.defaults), import contracts, license headers and the product guardrail. Tests: tests/profiler, tests/fluid_sql, the quality engine's column rules, the lazy command table and bootstrap. Each commit passes all of them. Its tests show only the three failures this shared environment has on the base too: two lazy-command import probes that read the main checkout's install, and DuckDB 1.5.1's reading of an infinite date.
  • Full suite once, on the earlier head that still held children per parent, with every model API key unset (-n 8, tests/perf and the compat gate run apart): 42,511 passed, 13 failed. Every failure is environmental. 11 fail the same way on a pristine copy of the base. The other 2 pass when run alone on the base and on this branch: a port already in use under xdist, and a hypothesis slow-generation health check under load. Run on their own: the compat gate (52 passed) and tests/perf (28 passed, the startup-module ceilings included; two wall-clock budgets failed with the machine's load above 100).
  • Secret scan of the touched files with detect-secrets 1.5.0: clean.
  • At this head: every commit's gates pass again, and its tests show 2,580 passed, with only the three baseline failures.
  • Live run on a three-expose 0.7.7 contract (enum, integer, decimal and date columns, two foreign keys, CSV and Parquet), with no --bundle:
    • one 1-1-0 document holding three tables;
    • the segment level histogram withholds the empty Edu level, and Ent beside it (C);
    • the seats range histogram runs on bins 1, 10, ..., 100, 100.5;
    • quantile/2 is published on dates and amounts;
  • fluxsim init from flux-sim-engine 0.7.0 (pypi.org, scratch venv) on that contract and profile exits 0 and writes its four-file bundle.

Borrow

Checked before building:

  • Month arithmetic: PostgreSQL + interval, DuckDB, BigQuery DATE_ADD, Snowflake DATEADD, Spark and Databricks add_months, Trino and SQL Server DATEADD clamp the day, while Oracle's, Snowflake's and Db2's ADD_MONTHS keep the month end. sqlglot 30.33 rewrites ADD_MONTHS faithfully only into DuckDB, so every dialect is rendered by hand.
  • Output checking for bins: the Eurostat and DwB output-checking guidelines (frequency tables at 10 units; minima, maxima and percentiles unsafe below that), ACRO 1.1.0, tau-ARGUS and sdcTable for secondary suppression, and SDMX CONF_STATUS for the codes. That led to data-independent grids through check_edge_counts.
  • CSV dates: DuckDB's CSV sniffer reads infinity and epoch as dates and drops a time of day, so a YYYY-MM-DD text is read by the profiler, strictly, instead.
  • Exact statistics: DuckDB aggregates, Polars, PyDeequ, Great Expectations and DataSketches. flux-stats already holds the registry, so nothing new is added.

Pages fetched: duckdb.org CSV overview, faulty CSV files, date and interval functions, date type and CASE; postgresql.org date/time and conditional functions; BigQuery date functions and conditional expressions; Snowflake DATEADD, ADD_MONTHS and CASE; Spark SQL functions, datetime functions, ANSI compliance and the 3.0 migration guide; Trino date/time and conditional functions; SQL Server DATEADD; Oracle 19c ADD_MONTHS; IBM Db2 ADD_MONTHS and date arithmetic; DuckDB's CSV sniffer sources.

No new dependency in core.

Follow-ups

  • forge_docs pages for whole-product profiles and default statistics.
  • The engine's fluxsim init does not read quantile/2 yet, and it should not generate past a contract's own bound (range end bins are rounded outward to the grid).
  • A date column in an NDJSON slice is still read as text.
  • Grouped number histograms are not declared yet.
  • Under --bundle: a Simulation block with narrow-bin histograms keeps sum, min, max and mean/1 beside them, and a block may still declare a histogram and quantile/2 on one column (a check_edge_counts question for flux-stats). nullCount is published as A.
  • The non-DuckDB renderings were checked by reading only. Running the vectors against Snowflake, BigQuery, Trino and Spark in CI would prove them.

… source

argparse kept the last of two --expose flags, so
`--expose org --expose account --source org.csv` profiled org.csv as the
account expose. A second --expose is now refused with
profile_expose_repeated before any record is read.
Amend decision 14 of RFC-data-rules.md: FLUID SQL/1 gains ADD_DAYS(d, n),
ADD_MONTHS(d, n) and ADD_YEARS(d, n), with flux-expr/3's meaning (flux-engine
ADR 0013, EXPR-SPEC-3 section 4). A month or year move clamps the day to the
last day of a shorter target month (31 Jan + 1 month = 28 or 29 Feb, and
28 Feb + 1 month = 28 Mar), a NULL argument gives NULL, and a result outside
0001-01-01..9999-12-31 is NULL, never an error. The addition is made in place
while 0.7.7 is a preview and fails closed: an older reader refuses the
functions as unknown.

Section 2.8 gives each dialect's rendering with a guarded count, so no engine
adds an out-of-range count: DuckDB to_months, PostgreSQL interval, BigQuery
DATE_ADD, Snowflake DATEADD (never its ADD_MONTHS, which keeps the month
end), Databricks add_months, Trino interval arithmetic, SQL Server DATEADD,
and month-end-safe forms for Oracle and Db2, whose ADD_MONTHS keeps the month
end. sqlglot is never used to transpile ADD_MONTHS: 30.33 rewrites it
faithfully only into DuckDB and passes it to other dialects by name.

Also correct section 1.4: fluid profile does not measure invariants yet.
…ole units

FLUID SQL/1 gains the three date moves the amended RFC section 2.6 names,
with flux-expr/3's meaning (flux-engine ADR 0013, EXPR-SPEC-3 section 4):

- the lexer's FUNCTIONS, the parser's call, a date_add node whose JSON form is
  EXPR-SPEC-3's ({node: date_add, unit, date, count}), its canonical text;
- the checker takes a date and a count of static scale 0, and refuses the
  NULL literal as either argument;
- the evaluator is exact and total: month and year moves clamp the day, and a
  result outside 0001-01-01..9999-12-31 is NULL for any count, 10^30 included;
- every dialect renders the engine's own date addition (DuckDB to_months,
  PostgreSQL interval, Snowflake DATEADD, BigQuery DATE_ADD, Databricks
  add_months, Trino interval arithmetic) on a count a CASE has made NULL where
  the result would leave the range, so no engine adds an overflowing count;
- a CEL rule refuses them, as it refuses the date differences.

fluid-sql-vectors.json is version 1.1.0, with 51 new vectors: month ends in
leap and common years, the first and last day of the range, huge counts,
NULL operands, and one vector per refusal. DuckDB computes every one of them.
The 0.7.7 schema is unchanged: it names FLUID SQL/1 and lists no function.
@github-actions github-actions Bot added provider Changes or requests related to providers cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes needs-docs Pull request needs a linked docs update or justification labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

  • Link a docs PR — check the "Docs PR linked" box and paste the URL
  • Mark as no docs needed — check "No docs needed" with a justification
  • Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.

--expose repeats, and each expose takes its records as
--source <exposeId>=<location> (a database's table as
--table <exposeId>=<table>). Without --expose, every expose a source names
is profiled, in the contract's order. One call writes one profile document
whose tables hold every expose, named after the contract's id unless --out
names it. The single-expose form is unchanged, byte for byte.

runner.profile_exposes forms every table from its own read, checks every
plan and block before any record is read, and has the registry read the
whole document back. profile_expose is that call for one expose. With
--bundle, each expose answers its own block, and the blocks of one document
must declare one grid.

Pairing mistakes are refused before any record is read, with stable events:
profile_source_unpaired, profile_source_missing, profile_source_unused,
profile_source_repeated, profile_table_unpaired, profile_expose_repeated.
A database source is masked in every refusal, after its exposeId or a
mistyped one.
A column the contract types date now reads a YYYY-MM-DD text as that day's
midnight in UTC, which is how a Parquet or database date already arrives.
A CSV holds every value as text, and the registry reads a timestamp only
from RFC 3339 with an offset, so a CSV date column was refused with
FLX-5022 while the same column from Parquet was read.

The text is read strictly, as the contract's type says: exactly YYYY-MM-DD
and a real calendar day. Anything else goes to the registry's reader as it
came, which refuses it. DuckDB's typed CSV reader is not used for this: it
reads infinity and epoch as dates and drops a time of day. A timestamp
column still needs RFC 3339.
Without --bundle, fluid profile now declares a fidelity block from the
contract alone, never from the records, and answers it through the same path
as a Simulation's block: flux-stats' grammar parses it, and every statistic
passes check_edge_counts (FLUX ADR 0012) and the ACRO rules. There is no
second parser.

- every enum(...) column: levelHistogram/1 over every declared level; a level
  no record holds is listed, its cell withheld as C;
- a number whose validationRules declare both ends of a range inside its
  type: marginalHistogram/1 over about ten bins of one width, exact at both
  ends (an integer range ends half a unit past its top, a whole unit at
  --fractional-bits 0, so each value has a bin of its own);
- any other number of a bounded type, a range of one side or with a side at
  the type's own bound included (ten bins over the rest of the type would
  hold every record in one), and every date and timestamp: quantile/2 at
  0.25, 0.5 and 0.75 on a ladder fixed by the type (0, 1, 2, 5, 10 ... within
  int64 or decimal(p,s); one bin per year from 1900 to 2100);
- each categorical semantics.dimensions entry on an enum column groups the
  other enum columns' level histograms, every declared level a group
  (DeclaredBlock.every_declared_group), so the groups never depend on
  records; one column's groupings hold at most 10,000 cells together, since
  checking them costs about the cube of the levels.

A record outside what the contract declares refuses the profile
(profile_record_outside_declared_domain), naming the column and row only.
The bins of a range are rounded outward to the grid, which can be coarser
than the column's scale, so each value is first checked against the range
itself, a strict bound included (DeclaredRange). A profile that holds
quantile/2 names format 1-1-0, so a profile without --bundle of a bounded
number, a date or a timestamp is now a 1-1-0 document.

tests/profiler/test_disclosure_oracle.py profiles every world of at most
seven records for each kind of table under scaled-down ACRO rules, groups
worlds by the document they publish, and fails on a protected count (one
record, or none) the new statistics pin. On larger universes it found four
channels, now closed:

- a column's distinctCount says how many withheld cells are empty: a column
  a level histogram counts, or a grouping groups by, withholds it (D), with or
  without --bundle;
- a number's exact sum, min, max, quantiles and distinctCount give a
  withheld bin of a declared range back (224 pins in 12,341 worlds): a number
  with a default histogram withholds them (D) and has no mean/1;
- a histogram and quantiles on one set of bins, checked together, let a
  reader who knows the check's choices tell worlds apart: a column gets one;
- a grouped statistic's sampleSize banded the records its groups hold, which
  beside the columns' own counts pinned the records that hold a value and a
  null key, under rules that round and under exact counts, once both columns
  held nulls: it now bands the column's non-null records, on --bundle too.

After the fixes, the ACRO defaults pin nothing: 17,296 worlds for a level
histogram, 46,376 for a grouping, 12,341 for a declared range, and 23,751
each for an integer and a date on their ladders. With nulls in both grouped
columns, none of 24,310 worlds of up to eight records pins one under any of
the audit's four rules. On a ladder, a histogram would publish nothing (its
empty end bins tie no cell to the range), so numbers there get quantiles and
keep their sum and mean/1.

The fake registry hands the fidelity grammar to flux-stats, and every
document to flux-stats' reader. fluid_build.profiler.defaults joins the
mypy --strict allowlist.
@fas89
fas89 force-pushed the claude/profile-sql-dates-6decaf branch from c13419c to 62fba0e Compare November 19, 2026 19:22
Comment thread fluid_build/profiler/disclosure.py Fixed
@fas89
fas89 force-pushed the claude/profile-sql-dates-6decaf branch from 62fba0e to f7dcfcf Compare November 19, 2026 19:41
@fas89
fas89 merged commit 1d7f6dc into main Nov 19, 2026
43 checks passed
@fas89
fas89 deleted the claude/profile-sql-dates-6decaf branch November 19, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli Changes to the CLI surface or implementation needs-docs Pull request needs a linked docs update or justification provider Changes or requests related to providers tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant