Repository navigation
feat(profile): whole-product profiles with default statistics, and FLUID SQL/1 date moves - #1024
Merged
Merged
Conversation
… source argparse kept the last of two --expose flags, so `--expose org --expose account --source org.csv` profiled org.csv as the account expose. A second --expose is now refused with profile_expose_repeated before any record is read.
Amend decision 14 of RFC-data-rules.md: FLUID SQL/1 gains ADD_DAYS(d, n), ADD_MONTHS(d, n) and ADD_YEARS(d, n), with flux-expr/3's meaning (flux-engine ADR 0013, EXPR-SPEC-3 section 4). A month or year move clamps the day to the last day of a shorter target month (31 Jan + 1 month = 28 or 29 Feb, and 28 Feb + 1 month = 28 Mar), a NULL argument gives NULL, and a result outside 0001-01-01..9999-12-31 is NULL, never an error. The addition is made in place while 0.7.7 is a preview and fails closed: an older reader refuses the functions as unknown. Section 2.8 gives each dialect's rendering with a guarded count, so no engine adds an out-of-range count: DuckDB to_months, PostgreSQL interval, BigQuery DATE_ADD, Snowflake DATEADD (never its ADD_MONTHS, which keeps the month end), Databricks add_months, Trino interval arithmetic, SQL Server DATEADD, and month-end-safe forms for Oracle and Db2, whose ADD_MONTHS keeps the month end. sqlglot is never used to transpile ADD_MONTHS: 30.33 rewrites it faithfully only into DuckDB and passes it to other dialects by name. Also correct section 1.4: fluid profile does not measure invariants yet.
…ole units
FLUID SQL/1 gains the three date moves the amended RFC section 2.6 names,
with flux-expr/3's meaning (flux-engine ADR 0013, EXPR-SPEC-3 section 4):
- the lexer's FUNCTIONS, the parser's call, a date_add node whose JSON form is
EXPR-SPEC-3's ({node: date_add, unit, date, count}), its canonical text;
- the checker takes a date and a count of static scale 0, and refuses the
NULL literal as either argument;
- the evaluator is exact and total: month and year moves clamp the day, and a
result outside 0001-01-01..9999-12-31 is NULL for any count, 10^30 included;
- every dialect renders the engine's own date addition (DuckDB to_months,
PostgreSQL interval, Snowflake DATEADD, BigQuery DATE_ADD, Databricks
add_months, Trino interval arithmetic) on a count a CASE has made NULL where
the result would leave the range, so no engine adds an overflowing count;
- a CEL rule refuses them, as it refuses the date differences.
fluid-sql-vectors.json is version 1.1.0, with 51 new vectors: month ends in
leap and common years, the first and last day of the range, huge counts,
NULL operands, and one vector per refusal. DuckDB computes every one of them.
The 0.7.7 schema is unchanged: it names FLUID SQL/1 and lists no function.
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
--expose repeats, and each expose takes its records as --source <exposeId>=<location> (a database's table as --table <exposeId>=<table>). Without --expose, every expose a source names is profiled, in the contract's order. One call writes one profile document whose tables hold every expose, named after the contract's id unless --out names it. The single-expose form is unchanged, byte for byte. runner.profile_exposes forms every table from its own read, checks every plan and block before any record is read, and has the registry read the whole document back. profile_expose is that call for one expose. With --bundle, each expose answers its own block, and the blocks of one document must declare one grid. Pairing mistakes are refused before any record is read, with stable events: profile_source_unpaired, profile_source_missing, profile_source_unused, profile_source_repeated, profile_table_unpaired, profile_expose_repeated. A database source is masked in every refusal, after its exposeId or a mistyped one.
A column the contract types date now reads a YYYY-MM-DD text as that day's midnight in UTC, which is how a Parquet or database date already arrives. A CSV holds every value as text, and the registry reads a timestamp only from RFC 3339 with an offset, so a CSV date column was refused with FLX-5022 while the same column from Parquet was read. The text is read strictly, as the contract's type says: exactly YYYY-MM-DD and a real calendar day. Anything else goes to the registry's reader as it came, which refuses it. DuckDB's typed CSV reader is not used for this: it reads infinity and epoch as dates and drops a time of day. A timestamp column still needs RFC 3339.
Without --bundle, fluid profile now declares a fidelity block from the contract alone, never from the records, and answers it through the same path as a Simulation's block: flux-stats' grammar parses it, and every statistic passes check_edge_counts (FLUX ADR 0012) and the ACRO rules. There is no second parser. - every enum(...) column: levelHistogram/1 over every declared level; a level no record holds is listed, its cell withheld as C; - a number whose validationRules declare both ends of a range inside its type: marginalHistogram/1 over about ten bins of one width, exact at both ends (an integer range ends half a unit past its top, a whole unit at --fractional-bits 0, so each value has a bin of its own); - any other number of a bounded type, a range of one side or with a side at the type's own bound included (ten bins over the rest of the type would hold every record in one), and every date and timestamp: quantile/2 at 0.25, 0.5 and 0.75 on a ladder fixed by the type (0, 1, 2, 5, 10 ... within int64 or decimal(p,s); one bin per year from 1900 to 2100); - each categorical semantics.dimensions entry on an enum column groups the other enum columns' level histograms, every declared level a group (DeclaredBlock.every_declared_group), so the groups never depend on records; one column's groupings hold at most 10,000 cells together, since checking them costs about the cube of the levels. A record outside what the contract declares refuses the profile (profile_record_outside_declared_domain), naming the column and row only. The bins of a range are rounded outward to the grid, which can be coarser than the column's scale, so each value is first checked against the range itself, a strict bound included (DeclaredRange). A profile that holds quantile/2 names format 1-1-0, so a profile without --bundle of a bounded number, a date or a timestamp is now a 1-1-0 document. tests/profiler/test_disclosure_oracle.py profiles every world of at most seven records for each kind of table under scaled-down ACRO rules, groups worlds by the document they publish, and fails on a protected count (one record, or none) the new statistics pin. On larger universes it found four channels, now closed: - a column's distinctCount says how many withheld cells are empty: a column a level histogram counts, or a grouping groups by, withholds it (D), with or without --bundle; - a number's exact sum, min, max, quantiles and distinctCount give a withheld bin of a declared range back (224 pins in 12,341 worlds): a number with a default histogram withholds them (D) and has no mean/1; - a histogram and quantiles on one set of bins, checked together, let a reader who knows the check's choices tell worlds apart: a column gets one; - a grouped statistic's sampleSize banded the records its groups hold, which beside the columns' own counts pinned the records that hold a value and a null key, under rules that round and under exact counts, once both columns held nulls: it now bands the column's non-null records, on --bundle too. After the fixes, the ACRO defaults pin nothing: 17,296 worlds for a level histogram, 46,376 for a grouping, 12,341 for a declared range, and 23,751 each for an integer and a date on their ladders. With nulls in both grouped columns, none of 24,310 worlds of up to eight records pins one under any of the audit's four rules. On a ladder, a histogram would publish nothing (its empty end bins tie no cell to the range), so numbers there get quantiles and keep their sum and mean/1. The fake registry hands the fidelity grammar to flux-stats, and every document to flux-stats' reader. fluid_build.profiler.defaults joins the mypy --strict allowlist.
fas89
force-pushed
the
claude/profile-sql-dates-6decaf
branch
from
November 19, 2026 19:22
c13419c to
62fba0e
Compare
fas89
force-pushed
the
claude/profile-sql-dates-6decaf
branch
from
November 19, 2026 19:41
62fba0e to
f7dcfcf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
fluid profilecan profile a whole data product in one call, and without--bundleit now publishes the statisticsfluxsim initreads, declared from the contract alone and checked like a Simulation's fidelity block. FLUID SQL/1 gains whole-unit date moves. Six commits, each green on its own:fix(profile): a repeated--exposeis refused (profile_expose_repeated) instead of profiling one expose's records as another's.docs(rfc):RFC-data-rules.mddecision 14 is amended. FLUID SQL/1 gainsADD_DAYS,ADD_MONTHSandADD_YEARSwith flux-expr/3's meaning: the day clamps to the end of a shorter month, NULL in gives NULL out, and a result outside 0001-01-01..9999-12-31 is NULL. It adds a per-dialect rendering table and a note that sqlglot is never used to transpileADD_MONTHS. Section 1.4 is corrected:fluid profiledoes not measure invariants yet.feat(fluid-sql): the lexer, parser, checker, exact evaluator, canonical text, the six dialect renderers (each on a count aCASEhas made NULL where the result would leave the range) and the CEL refusal. Conformance vectors are 1.1.0, with 51 new vectors.fluid-schema-0.7.7.jsonis byte-identical.feat(profile):--exposerepeats, each paired with--source <exposeId>=<location>(and--table <exposeId>=<table>). One call writes one document holding every expose. One--exposewith one bare--sourceworks as before.fix(profile): a CSVdatecolumn reads itsYYYY-MM-DDtext as that day.feat(profile): default statistics without--bundle(below).Children per parent for each
semantics.relationshipsentry moves to a follow-up PR.Default statistics
The profiler writes its own fidelity block from the contract and answers it through the
--bundlepath, so flux-stats' grammar parses it. Every statistic goes throughcheck_edge_counts(ADR 0012) and the ACRO rules. No bin edge ever comes from the data.enum(...)levelHistogram/1over every declared level; a level no record holds is a withheld cell (C)validationRulesclose a range inside its typemarginalHistogram/1, about ten bins of one width; each value is checked exactly against the rangequantile/2at 0.25, 0.5 and 0.75 on a ladder fixed by the type (1-2-5 steps within the type; one bin per year 1900 to 2100)semantics.dimensionsentryChoices the disclosure audit forced:
sum,min,max,quantilesanddistinctCount(D) and forms nomean/1, since each gave a withheld bin back;A record outside what the contract declares refuses the profile (
profile_record_outside_declared_domain, naming the column and row only). This is deliberate: dropping just that column's statistic would make the document's shape depend on the records.A profile holding
quantile/2names format 1-1-0, so most profiles without--bundleare 1-1-0 documents now. flux-stats reads them from 0.7, and the[profile]pin staysflux-stats>=0.7,<0.8.Disclosure audit
tests/profiler/test_disclosure_oracle.pyenumerates every world (a multiset of records) of tiny instances: 1 or 2 tables, at most 3 bins, at most 7 rows. It runs the real profiler on each world, groups the worlds by the canonical bytes they publish (so the publish decisions are part of what a reader sees), and fails when a protected count (below the minimum cell, or a zero) is the same in every world of a group but was not already fixed without the new statistics.Larger runs (not in the suite):
The audit and the reviews found four channels, all closed here:
distinctCountsays how many withheld cells are empty, so a column a level histogram counts, or a grouping groups by, withholds it (D), with or without--bundle;sum,min,max,quantilesanddistinctCountgave a withheld bin of a range back (224 pins in 12,341 worlds before the fix);sampleSizebanded the records its groups hold. Beside the columns' own counts, that fixed the number of records with a value and a null key. It now bands the column's non-null records, on--bundletoo.Reviews
sampleSizeband). After the fix the reviewer re-ran its own audit (24,310 worlds, 0 pins) and gave SHIP.validationRules(it crashed);--fractional-bits 0;Each new regression test fails on the pre-fix head and passes here.
Validation
scripts/mypy_strict.py(the strict allowlist now includesfluid_build.profiler.defaults), import contracts, license headers and the product guardrail. Tests:tests/profiler,tests/fluid_sql, the quality engine's column rules, the lazy command table and bootstrap. Each commit passes all of them. Its tests show only the three failures this shared environment has on the base too: two lazy-command import probes that read the main checkout's install, and DuckDB 1.5.1's reading of an infinite date.-n 8,tests/perfand the compat gate run apart): 42,511 passed, 13 failed. Every failure is environmental. 11 fail the same way on a pristine copy of the base. The other 2 pass when run alone on the base and on this branch: a port already in use under xdist, and a hypothesis slow-generation health check under load. Run on their own: the compat gate (52 passed) andtests/perf(28 passed, the startup-module ceilings included; two wall-clock budgets failed with the machine's load above 100).--bundle:segmentlevel histogram withholds the emptyEdulevel, andEntbeside it (C);seatsrange histogram runs on bins1, 10, ..., 100, 100.5;quantile/2is published on dates and amounts;fluxsim initfrom flux-sim-engine 0.7.0 (pypi.org, scratch venv) on that contract and profile exits 0 and writes its four-file bundle.Borrow
Checked before building:
+ interval, DuckDB, BigQueryDATE_ADD, SnowflakeDATEADD, Spark and Databricksadd_months, Trino and SQL ServerDATEADDclamp the day, while Oracle's, Snowflake's and Db2'sADD_MONTHSkeep the month end. sqlglot 30.33 rewritesADD_MONTHSfaithfully only into DuckDB, so every dialect is rendered by hand.CONF_STATUSfor the codes. That led to data-independent grids throughcheck_edge_counts.infinityandepochas dates and drops a time of day, so aYYYY-MM-DDtext is read by the profiler, strictly, instead.Pages fetched: duckdb.org CSV overview, faulty CSV files, date and interval functions, date type and CASE; postgresql.org date/time and conditional functions; BigQuery date functions and conditional expressions; Snowflake
DATEADD,ADD_MONTHSandCASE; Spark SQL functions, datetime functions, ANSI compliance and the 3.0 migration guide; Trino date/time and conditional functions; SQL ServerDATEADD; Oracle 19cADD_MONTHS; IBM Db2ADD_MONTHSand date arithmetic; DuckDB's CSV sniffer sources.No new dependency in core.
Follow-ups
fluxsim initdoes not readquantile/2yet, and it should not generate past a contract's own bound (range end bins are rounded outward to the grid).--bundle: a Simulation block with narrow-bin histograms keepssum,min,maxandmean/1beside them, and a block may still declare a histogram andquantile/2on one column (acheck_edge_countsquestion for flux-stats).nullCountis published as A.