Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ forge-cli/
| Bundle builder | `fluid_build/forge/core/bundle.py` — deterministic tgz + MANIFEST + fragment extraction + `$source` sentinel |
| Build runners | `fluid_build/build_runners/{dbt,python}/` — extracted from legacy `cli/execute.py` in Phase 1 |
| Masking at landing | `fluid_build/build_runners/_masking.py` — the DuckDB runner applies `exposes[].policy.privacy.masking[]` inside the COPY (file, S3, BigQuery staging, DLQ); secrets only from env (`FLUID_PII_HASH_SECRET`, `FLUID_PII_TOKENIZATION_KEY`, `FLUID_PII_ENCRYPTION_SECRET_KEY`, or `params.saltEnv`/`keyEnv`), held in UDF closures, never in SQL; an unappliable rule refuses the build, and so does a DuckDB without numpy (its `create_function` needs it; `[local]` names numpy, and the refusal names the extra). `cli/_verify_masking.py` is stage 9's shape check (local DuckDB, Athena in the count query) |
| FLUX profiler (`fluid profile`, `[profile]` extra) | `fluid_build/profiler/`: reads one expose's records where they live (`sources.py`: an NDJSON slice as flux-engine reads one; CSV/Parquet and postgres/mysql/sqlite through the DuckDB sandbox and `cli/discover/_jdbc_introspect.attach_jdbc_source`), applies ACRO's output checks (`disclosure.py`) and writes a PROFILE-SPEC 1-0-0 document (`document.py`, RFC 8785, digest `sha256:`): the OpenLineage column metrics, plus an ungrouped `mean/1` per number column on the document's grid (`--fractional-bits`, default 16), published only where the column's `count` and `sum` are (`disclosure.check_mean`). With `--bundle DIR` it also answers the fidelity block of DIR's one Simulation: `bundle.py` reads `*.flux.yml` as the engine does (its exact YAML loader, then KERNEL-SPEC 2.1 canonical form), flux-stats' `flux_stats.fidelity` parses the block (no parser here), and each declared property's statistic is formed per group on the block's grid, every group and cell output-checked; a withheld group of a grouped mean or support withholds its siblings under `D`, and a correlation is not answered. Every value is read, and every statistic formed, by FLUX's registry `flux-stats` through the one seam `_stats.py`, which imports it only when a profile runs; never import `flux_engine` here (`tests/profiler/test_boundary.py`) |
| FLUX profiler (`fluid profile`, `[profile]` extra) | `fluid_build/profiler/`: reads one expose's records where they live (`sources.py`: an NDJSON slice as flux-engine reads one; CSV/Parquet and postgres/mysql/sqlite through the DuckDB sandbox and `cli/discover/_jdbc_introspect.attach_jdbc_source`), applies ACRO's output checks (`disclosure.py`) and writes a PROFILE-SPEC 1-0-0 document (`document.py`, RFC 8785, digest `sha256:`): the OpenLineage column metrics, plus an ungrouped `mean/1` per number column on the document's grid (`--fractional-bits`, default 16), published only where the column's `count` and `sum` are (`disclosure.check_mean`). With `--bundle DIR` it also answers the fidelity block of DIR's one Simulation: `bundle.py` reads `*.flux.yml` as the engine does (flux-stats' `flux_stats.exact_yaml` and `flux_stats.canonical`, through the seam, one `ReadBudget` per bundle), flux-stats' `flux_stats.fidelity` parses the block (no parser here), and each declared property's statistic is formed per group on the block's grid, every group and cell output-checked; a quantile with bins is `quantile/2` (the profile then names format 1-1-0), the histograms and quantiles of one column go to flux-stats' `check_edge_counts` in one call (FLUX ADR 0012; `runner._edge_decisions`), a withheld group of a grouped mean or support withholds its siblings under `D`, and a correlation is not answered. Every value is read, and every statistic formed, by FLUX's registry `flux-stats` through the one seam `_stats.py`, which imports it only when a profile runs; never import `flux_engine` here (`tests/profiler/test_boundary.py`) |
| CI template generator | `fluid_build/forge/core/pipeline_templates.py` — 7-system generator; Jenkins ships the full 11-stage parameterized template with `--install-mode {pypi,dev-source}` |
| Forge (AI creation) | `fluid_build/forge/` (templates, generators, extensions) |
| Log redaction (global) | `fluid_build/observability/secret_redactor.py` — `SecretRedactingFilter` wired into Python logging |
Expand Down
50 changes: 50 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **`fluid profile --bundle` publishes a quantile with bins as its bin.** A `quantile` property
that declares `bins` is now answered with `quantile/2`: the group states the declared bin that
holds the order statistic at its level, and the records below that bin and through it, and no
witness. A reader checks that the rounded counts still put the order statistic in that bin,
and a group whose rounded counts do not is withheld. A profile that holds one names format
`1-1-0`; every other profile keeps `1-0-0` and its bytes. A quantile with bins is published
beside a histogram that shares its bins, where a `quantile/1` is withheld.

### Changed

- **`fluid profile --bundle` reads a bundle with flux-stats.** The bundle's YAML is read by
flux-stats' exact reader and its block put in the canonical form by flux-stats, instead of a
copy of the engine's code in forge-cli. The bounds hold as before (aliases, merge keys, digits
written out, nesting depth, document size, one budget across a bundle's files), and a refusal
still names the file, and the line where the reader reports one, never a value. Its words
differ in places: a YAML error names the line and column where the reader found it, the most
values a document may write out through its aliases is now 1,000,000, and the characters it
writes out are bounded too. A block whose bins are not strictly increasing grid points is
refused before any record is read (`FLX-5004`), as the fidelity check refuses it. The
`[profile]` extra needs `flux-stats[yaml]>=0.7.0,<0.8`: 0.7.0 is the first flux-stats release
that holds these readers and the check below.

### Security

- **A histogram, a level histogram and a quantile of one column are checked together (FLUX ADR
0012).** 0.33.0's group-size, quantile and sibling checks were each one part of a rule, and a
reader could still subtract one histogram's published counts from another's, or from a level
histogram's group size, and rebuild a withheld cell, a zero included. `fluid profile` now
passes every histogram, level histogram and quantile of a column to flux-stats'
`check_edge_counts` in one call, and publishes what it lets through. More is withheld:
- differencing between histograms: a histogram and a coarser one of one column, or a published
cell and the column's count, no longer give a withheld cell back by subtraction, and a cell
that does goes with it, counted as `D`;
- a level histogram is checked over its levels as it is over bins, so one small level
withholds a neighbour;
- a zero cell a reader can form, where zeros are disclosive: cells of 12, 0 and 12 records
withhold one of the 12s with the zero;
- a histogram that does not sit on the column's one partition of bins, and a group whose
rounded counts no longer prove its quantile;
- a group that is withheld whole beside rows no group holds that the block cannot read.

The summary counts a cell or group withheld for its size or rounding as `A`, and one withheld
so that another cannot be recovered as `D`. A group whose every cell is withheld is still
withheld whole. The check does not see a column's `support/1`, which lists the levels that hold
records, the group sizes of a statistic grouped by the column, which repeat its level counts,
or a declared level that holds no record, which is absent from a grouped statistic; `fluid
profile --help` says so.

## [0.34.0] — 2026-11-19

### Added
Expand Down
63 changes: 40 additions & 23 deletions fluid_build/cli/profile_cmd.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@
"""``fluid profile``: an output-checked profile of one expose, read where its records live.

The command reads the expose's records once, inside the environment that holds
them, and writes a FLUX profile document (PROFILE-SPEC format 1-0-0) of
aggregates that passed the disclosure rules. A FLUX Simulation pins the document
by the digest this prints. See :mod:`fluid_build.profiler` for the parts.
them, and writes a FLUX profile document (PROFILE-SPEC format 1-0-0, or 1-1-0
when it holds a quantile with bins) of aggregates that passed the disclosure
rules. A FLUX Simulation pins the document by the digest this prints. See
:mod:`fluid_build.profiler` for the parts.

Registered lazily (``cli/bootstrap.py`` ``CORE_COMMANDS``), and only the standard library is
imported at module scope, so neither ``fluid --help`` nor ``fluid profile
Expand All @@ -26,15 +27,15 @@
_DESCRIPTION = """\
Write an output-checked profile of one expose's records, read where they live.

The document follows FLUX PROFILE-SPEC (format 1-0-0) and holds only aggregates
that passed the output checks: no record, no sample, no per-record value. The
checks are ACRO 1.1.0's defaults, named "acro-default" in the document:
The document follows FLUX PROFILE-SPEC (format 1-0-0, or 1-1-0 when it holds a
quantile with bins) and holds only aggregates that passed the output checks: no
record, no sample, no per-record value. The checks are ACRO 1.1.0's defaults, named
"acro-default" in the document:
- a count below 10 records is withheld, and so is a zero;
- every published count is rounded to a multiple of 5;
- a sum is withheld when 2 records make up 90% of it, or when what is left
after the 2 largest is below 10% of the largest;
- a min, max or quantile is published only when 10 records hold that value,
and a quantile only when at least 10 records lie below it and 10 above it;
- a min, max or quantile is published only when 10 records hold that value;
- a mean is published only when the column's count and sum are.
A withheld value keeps an SDMX CL_CONF_STATUS code and no value: A (a small
count), O or T (dominated by one or two records), M (concentration), D
Expand Down Expand Up @@ -62,12 +63,26 @@
What the checks leave readable: a grouped statistic lists every group the data
forms, published or withheld. A key is always a level the bundle declares (a
value outside the declared support is refused), so its presence says only that
the level holds at least one record. A group withheld for its size withholds
every group of its groupBy's histograms and quantiles on its column too, and a
quantile is published only as the one histogram or quantile of its column. A withheld cell of
a histogram is not secondarily suppressed: the rounded counts still bound it,
inside 0 to 9 records, and never above the threshold. A published min or max
shows that no record lies below or above it. A quantile whose value is the
the level holds at least one record.

The counts a histogram, a level histogram or a quantile publishes at the edges
of one column are checked together, as FLUX ADR 0012 sets, by flux-stats'
check_edge_counts. They sit on one partition (the column's bins, or its
levels). A quantile that declares bins is written as quantile/2, with the bin
that holds its order statistic and the records below that bin and through it,
and the profile then names format 1-1-0; its three cells (below, in and above
the bin) each hold at least 10 records, or none at the ends of the range. A
quantile without bins is published only as the one such statistic of its column.
A group of fewer than 10 records withholds every group of its groupBy, and a
group whose every cell is withheld is withheld whole. Across those statistics
of one column, no small cell, a zero included where zeros are disclosive, can be
formed by adding or subtracting what they publish, so more is withheld (D in
the summary). Only then are counts rounded. The check does not see a column's
support, which lists the levels that hold records, or the group sizes of a
statistic grouped by the column, which repeat its level counts; and a declared
level that holds no record is absent from a grouped statistic, which shows its
zero. A published min or
max shows that no record lies below or above it. A quantile whose value is the
column's smallest or largest shows the same, and is withheld for it, as a zero
is.

Expand All @@ -86,14 +101,16 @@

# Only aggregates leave: a count under 10 or a zero is withheld, counts are rounded
# to 5, a sum 2 records dominate is withheld, and a min, max or quantile is shown only
# when 10 records hold it (ACRO 1.1.0 defaults), a quantile only when at least 10
# records lie below it and 10 above it, and only as its column's one histogram or
# quantile. A group of fewer than 10 records withholds every group of its groupBy's
# histograms and quantiles on its column. A histogram's or a quantile's withheld values all read C,
# whatever withheld them. A min or max is published though it shows that no record lies
# below or above it, and a quantile at the column's smallest or largest value is withheld
# for showing the same. Output-checking conventions only: no privacy or anonymity
# guarantee. Pin the printed digest in the FLUX Simulation.
# when 10 records hold it (ACRO 1.1.0 defaults). With --bundle, a column's histograms and
# quantiles are checked together (FLUX ADR 0012): on one partition, a quantile with bins
# only with at least 10 records below its bin, in it and above it, and no small cell,
# a zero included, left to be formed by adding or subtracting what is published. A group
# of fewer than 10 records withholds every group of its groupBy. A histogram's or a
# quantile's withheld values all read C, whatever withheld them. A min or max is published
# though it shows that no record lies below or above it, and a quantile at the column's
# smallest or largest value is withheld for showing the same. Output-checking
# conventions only: no privacy or anonymity guarantee. Pin the printed digest in the
# FLUX Simulation.
# Each number column also gets mean/1, which FLUX compares a mean property with, on the
# grid --fractional-bits names (16 unless given): the fidelity block must declare it.

Expand Down Expand Up @@ -216,7 +233,7 @@ def run(args: argparse.Namespace, logger: logging.Logger) -> int:
try:
expose_id = runner.resolve_expose_id(contract, args.expose_id)
out = _out_path(args.out, expose_id, args.source)
block = bundle.read_bundle_block(bundle_dir, expose_id) if bundle_dir else None
block = bundle.read_bundle_block(bundle_dir, expose_id, backend) if bundle_dir else None
result = runner.profile_expose(
contract,
expose_id,
Expand Down
Loading
Loading