Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ forge-cli/
| Bundle builder | `fluid_build/forge/core/bundle.py` — deterministic tgz + MANIFEST + fragment extraction + `$source` sentinel |
| Build runners | `fluid_build/build_runners/{dbt,python}/` — extracted from legacy `cli/execute.py` in Phase 1 |
| Masking at landing | `fluid_build/build_runners/_masking.py` — the DuckDB runner applies `exposes[].policy.privacy.masking[]` inside the COPY (file, S3, BigQuery staging, DLQ); secrets only from env (`FLUID_PII_HASH_SECRET`, `FLUID_PII_TOKENIZATION_KEY`, `FLUID_PII_ENCRYPTION_SECRET_KEY`, or `params.saltEnv`/`keyEnv`), held in UDF closures, never in SQL; an unappliable rule refuses the build, and so does a DuckDB without numpy (its `create_function` needs it; `[local]` names numpy, and the refusal names the extra). `cli/_verify_masking.py` is stage 9's shape check (local DuckDB, Athena in the count query) |
| FLUX profiler (`fluid profile`, `[profile]` extra) | `fluid_build/profiler/`: reads each expose's records where they live, every expose of one call into one document (`runner.profile_exposes`; the CLI pairs `--source <exposeId>=<location>` and `--table <exposeId>=<table>` with repeated `--expose`) (`sources.py`: an NDJSON slice as flux-engine reads one; CSV/Parquet and postgres/mysql/sqlite through the DuckDB sandbox and `cli/discover/_jdbc_introspect.attach_jdbc_source`), applies ACRO's output checks (`disclosure.py`) and writes a PROFILE-SPEC document, format 1-0-0 or, once it holds a `quantile/2`, 1-1-0 (`document.py`, RFC 8785, digest `sha256:`): the OpenLineage column metrics, plus an ungrouped `mean/1` per number column on the document's grid (`--fractional-bits`, default 16), published only where the column's `count` and `sum` are (`disclosure.check_mean`). Without `--bundle` it declares, from the contract alone, a fidelity block of the statistics `fluxsim init` reads (`defaults.py`: a `levelHistogram/1` over every level of an `enum(...)` column, a `marginalHistogram/1` over a `validationRules` range of two ends inside the type (each value checked against the range exactly, `DeclaredRange`), else `quantile/2` on a ladder fixed by the type for numbers and dates, grouped level histograms per categorical `semantics.dimensions` level, at most `defaults.MOST_GROUPED_CELLS` cells per column; `DeclaredBlock.every_declared_group` lists every declared level as a group) and answers it through the same path; `tests/profiler/test_disclosure_oracle.py` enumerates every tiny world to show no new statistic pins a protected count, which is why a column never gets a histogram and quantiles together and why `runner._counted_levels` / `_binned_numbers` withhold `distinctCount`, and a declared range's `sum`, `min`, `max` and `quantiles`. With `--bundle DIR` it answers the fidelity block of DIR's one Simulation instead: `bundle.py` reads `*.flux.yml` as the engine does (flux-stats' `flux_stats.exact_yaml` and `flux_stats.canonical`, through the seam, one `ReadBudget` per bundle), flux-stats' `flux_stats.fidelity` parses the block (no parser here), and each declared property's statistic is formed per group on the block's grid, every group and cell output-checked; a quantile with bins is `quantile/2` (the profile then names format 1-1-0), the histograms and quantiles of one column go to flux-stats' `check_edge_counts` in one call (FLUX ADR 0012; `runner._edge_decisions`), a withheld group of a grouped mean or support withholds its siblings under `D`, and a correlation is not answered. Every value is read, and every statistic formed, by FLUX's registry `flux-stats` through the one seam `_stats.py`, which imports it only when a profile runs; never import `flux_engine` here (`tests/profiler/test_boundary.py`) |
| FLUX profiler (`fluid profile`, `[profile]` extra) | `fluid_build/profiler/`: reads each expose's records where they live, every expose of one call into one document (`runner.profile_exposes`; the CLI pairs `--source <exposeId>=<location>` and `--table <exposeId>=<table>` with repeated `--expose`) (`sources.py`: an NDJSON slice as flux-engine reads one; CSV/Parquet and postgres/mysql/sqlite through the DuckDB sandbox and `cli/discover/_jdbc_introspect.attach_jdbc_source`), applies ACRO's output checks (`disclosure.py`) and writes a PROFILE-SPEC document, format 1-0-0 or, once it holds a `quantile/2`, 1-1-0 (`document.py`, RFC 8785, digest `sha256:`): the OpenLineage column metrics, plus an ungrouped `mean/1` per number column on the document's grid (`--fractional-bits`, default 16), published only where the column's `count` and `sum` are (`disclosure.check_mean`). Without `--bundle` it declares, from the contract alone, a fidelity block of the statistics `fluxsim init` reads (`defaults.py`: a `levelHistogram/1` over every level of an `enum(...)` column, a `marginalHistogram/1` over a `validationRules` range of two ends inside the type (each value checked against the range exactly, `DeclaredRange`), else `quantile/2` on a ladder fixed by the type for numbers and dates, grouped level histograms per categorical `semantics.dimensions` level, at most `defaults.MOST_GROUPED_CELLS` cells per column; `DeclaredBlock.every_declared_group` lists every declared level as a group) and answers it through the same path; `tests/profiler/test_disclosure_oracle.py` enumerates every tiny world to show no new statistic pins a protected count, which is why a column never gets a histogram and quantiles together and why `runner._counted_levels` / `_binned_numbers` withhold `distinctCount`, and a declared range's `sum`, `min`, `max` and `quantiles`. A `semantics.relationships` entry between two exposes of one call gets children per parent in the document's `relationships[]` (`runner._children_per_parent`, bins `runner.CHILDREN_EDGES`, the parent as unit, its own `check_edge_counts` call), and the child's foreign key withholds its `distinctCount`, `sum`, `min`, `max` and `quantiles` and forms no `mean/1`; `runner._audited_children` then withholds every cell the strongest reader could work out: the numbers of parents, children and null-key parents known exactly, the check's own rule included (`disclosure.pinned_children_cells`, a witness search that errs toward withholding); the key columns of a counted relationship take no default statistic (`runner._relationship_keys`), since a key's own histogram or quantiles would count each parent's children. With `--bundle DIR` it answers the fidelity block of DIR's one Simulation instead: `bundle.py` reads `*.flux.yml` as the engine does (flux-stats' `flux_stats.exact_yaml` and `flux_stats.canonical`, through the seam, one `ReadBudget` per bundle), flux-stats' `flux_stats.fidelity` parses the block (no parser here), and each declared property's statistic is formed per group on the block's grid, every group and cell output-checked; a quantile with bins is `quantile/2` (the profile then names format 1-1-0), the histograms and quantiles of one column go to flux-stats' `check_edge_counts` in one call (FLUX ADR 0012; `runner._edge_decisions`), a withheld group of a grouped mean or support withholds its siblings under `D`, and a correlation is not answered. Every value is read, and every statistic formed, by FLUX's registry `flux-stats` through the one seam `_stats.py`, which imports it only when a profile runs; never import `flux_engine` here (`tests/profiler/test_boundary.py`) |
| CI template generator | `fluid_build/forge/core/pipeline_templates.py` — 7-system generator; Jenkins ships the full 11-stage parameterized template with `--install-mode {pypi,dev-source}` |
| Forge (AI creation) | `fluid_build/forge/` (templates, generators, extensions) |
| Log redaction (global) | `fluid_build/observability/secret_redactor.py` — `SecretRedactingFilter` wired into Python logging |
Expand Down
32 changes: 32 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **`fluid profile` counts children per parent for each declared relationship.** When one call
profiles both exposes of a `semantics.relationships` entry, without `--bundle`, the
document's `relationships[]` (PROFILE-SPEC §5, format unchanged) carries a
`marginalHistogram/1` with the parent as unit: each parent row counts once, by its number of
child rows, on the bins 0, 1, 2, 3 to 4, 5 to 9, 10 to 19, and 20 or more. The last bin ends
at 2^53 - 1, the largest count a profile holds, so no count is cut to fit. Keys are matched
as the registry reads them, and a null key matches none. The histogram goes through
`check_edge_counts` in a call of its own, as flux-stats documents for a relationship's count
of children. The child's foreign-key columns withhold their `distinctCount`, `sum`, `min`,
`max` and `quantiles` under `D` and form no `mean/1`: the parents that have a child, beside
the parents' row count, would give those with none, and the key's own values describe each
parent's children key by key. A relationship whose target is not profiled in the call,
declares a validity window, or whose keys are read in different domains is named in the
summary and left out. The key columns
of a counted relationship take no default statistic of their own: a key's level histogram,
histogram or quantiles count each key's rows, and the child key's rows are each parent's
children. The check sees the parents' cells and their total, while a reader can know more:
other tables over the same parents (an `enum` column's level histogram, another
relationship's cells) bound their number, and the children's own counts bound what they
hold, each child row having at most one parent and each parent between its bin's edges.
So each withheld cell is audited against the strongest reader, who knows the number of
parents, of children and of parents whose key is null exactly, besides the published
bands and the check's own rule: a withheld cell stands only where two tables that reader
can hold true differ in it. While one cannot be shown to, the published cell of the
highest edge is withheld too, down to the whole histogram, which is where parents with
few children often land. Audited world by world with children unbounded and none an
orphan: under ACRO's rule no split of a band of children over a band of parents pins a
count (up to 144,962 worlds), and up to 7 parents keyed by their exact numbers pin none
under every scaled rule, exact counts included.

## [0.36.0] — 2026-11-19

### Added
Expand Down
23 changes: 22 additions & 1 deletion fluid_build/cli/profile_cmd.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,14 @@
date or a timestamp);
- for each categorical semantics.dimensions entry on an enum column, the
other enum columns' histograms per level of it, every declared level a
group, up to 10,000 cells for one column's groupings.
group, up to 10,000 cells for one column's groupings;
- for each semantics.relationships entry between two exposes of the call,
children per parent: how many child rows each parent row has, counted on
the bins 0, 1, 2, 3-4, 5-9, 10-19 and 20 or more, with the parent as the
unit; the child's foreign key then withholds its distinctCount, sum,
min, max and quantiles, its key columns take no statistic of their own,
and a cell a reader who knows the numbers of parents and children
exactly could work out is withheld too.
A record outside what the contract declares (a level its type does not list, a
value outside its type or its declared range) refuses the profile.

Expand Down Expand Up @@ -734,6 +741,20 @@ def _report(say: Any, result: Any, out: Path, *, answered_block: bool = False) -
)
for table in tables:
_report_table(say, table, f"exposes[{table.plan.expose_id}]: " if several else "", grid)
for counted in result.relationships:
link = counted.relationship
shown = (
f"{counted.published} of {counted.cells} cells published"
if counted.published
else "withheld whole"
)
say(
f"Children per parent of exposes[{link.parent}] in exposes[{link.child}] "
f"({link.name}): {shown}.",
soft_wrap=True,
)
for child, name, reason in result.relationships_left_out:
say(f"No children per parent for exposes[{child}] {name!r}: {reason}.", soft_wrap=True)
if not result.published:
say("Every value was withheld: the document says nothing about the records.")
say(f"Wrote {out} (readable by you alone; read it before you share it).", soft_wrap=True)
Expand Down
31 changes: 31 additions & 0 deletions fluid_build/profiler/_stats.py
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,15 @@ def measure(
record the block's declaration refuses.
"""

def count_histogram(
self, counts: Sequence[int], edges: Sequence[int], fractional_bits: int
) -> MeasuredGroup:
"""``marginalHistogram/1`` over whole numbers, one per unit: the ungrouped group.

*counts* are one value per unit (a parent's number of children), and *edges* the
bins' whole-number edges, placed on the grid as the registry places any edge.
"""

def check_edge_counts(
self,
statistics: Sequence[EdgeStatistic],
Expand Down Expand Up @@ -522,6 +531,7 @@ def __init__(self) -> None:
from flux_stats.reading import ValueDomain, category, read_exact

self._metrics = metrics
self._parameters = Parameters
self._statistics = tuple(_Statistic(each, Parameters()) for each in definitions())
self._grid = Grid
self._refused = FluxError
Expand Down Expand Up @@ -823,6 +833,27 @@ def _measured(
)
return replace(group, values=tuple(values), encoded=tuple(encoded))

def count_histogram(
self, counts: Sequence[int], edges: Sequence[int], fractional_bits: int
) -> MeasuredGroup:
grammar = self._fidelity()
grid = self._grid(fractional_bits)
histogram = grammar.definition_named("marginalHistogram/1")
encoded = tuple(grid.quantise(Fraction(edge), subject="bin edge") for edge in edges)
state = histogram.measure(
"",
[grid.quantise(Fraction(count), subject="count") for count in counts],
self._parameters(edges=encoded),
)
spelled: Dict[str, Any] = state.to_document(histogram)
return MeasuredGroup(
key="",
document=spelled,
k=state.k,
statistic=state,
cells=tuple(state.cell_counts("marginalHistogram")),
)

def check_edge_counts(
self,
statistics: Sequence[EdgeStatistic],
Expand Down
Loading
Loading