Repository navigation
feat(profile): check a column's edge counts with flux-stats 0.7.0 - #1021
Merged
Merged
Conversation
bundle.py kept its own port of the engine's exact loader and canonical form. It now reads each file with flux_stats.exact_yaml.load_documents, under one ReadBudget for the bundle, and canonicalises the block with flux_stats.canonical, both through the backend seam. The file listing, the per-file byte cap and the BundleError events stay. The [profile] extra and the install message name flux-stats[yaml]>=0.7.0,<0.8.
A profile names format 1-0-0 unless one of its statistics is quantile/2, which only format 1-1-0 carries, so every existing profile keeps its bytes. document.py learns the 1-1-0 URL and picks it from the tables it is given; a test keeps it equal to flux_stats.profile's.
…e quantile/2 A quantile that declares bins is quantile/2: its group states the bin that holds the order statistic and the records below and through it, with no witness. The backend seam gains check_edge_counts, and the runner passes every histogram, level histogram and quantile of a column to it in one call, with the rows a null groupBy key leaves out as one more group that is counted and never written. Retires the profiler's own copies of that rule: the sibling and crowded checks in the runner, the histogram and quantile branches of _check_group, publish_cells, check_quantile, written_status and the backend's quantile_ranks_ok. Mean and support keep their own checks and secondary suppression.
…ounts CHANGELOG and the --help text describe the one rule for histograms, level histograms and quantiles, the quantile/2 a quantile with bins is published as, format 1-1-0, and the bundle reader. The note that the earlier entries did not complete ADR 0012's rule is removed, since this completes it. AGENTS.md names the new seam calls.
…r can use flux-stats 0.7.0 is on PyPI, so the pin's comment and the install message's note name it as the first release that holds edge_counts, exact_yaml and quantile/2, instead of a placeholder for the maintainers to fill in.
…helpers A histogram's or a quantile's groups follow the output check on its column, and a mean's or a support's their own checks with their siblings. Each path is now a helper of its own, so the entry that writes a statistic only picks one.
… left, say only a refusal's code From the reviews of this branch: - Bins whose edges are not strictly increasing grid points (0.4 and 0.6 at fractionalBits 0) left the edge check with no partition, and flux-stats raised a KeyError the profile did not catch. Such a block is now refused before a record is read, by the registry's own rule (FLX-5004), as the fidelity check refuses it. - A histogram group the check let through with every cell withheld was written as published, with null cells, and the report named the statistic as answered. It is withheld whole again, as a group whose every cell is withheld was before the check ran on edges. - fluid profile --help said no small cell can be formed from what is published. It now says the check covers one column's histograms, level histograms and quantiles, and names what it does not see. - A refusal from inside check_edge_counts names its code, not the registry's words, which may give counts, and FLX-1007 says its title, not the colliding key.
…x-stats 0.7.0 Under the Unreleased section after 0.34.0: quantile/2 and format 1-1-0 (Added), the bundle read through flux-stats' exact reader, the FLX-5004 refusal of bins off the grid and the flux-stats 0.7.0 pin (Changed), and the joint check of a column's histograms, level histograms and quantiles, with what it does not see (Security). The released sections, #995's note in 0.33.0 included, are untouched.
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fluid profile --bundlenow uses the released flux-stats 0.7.0 for its output checks and its bundle reading. The profiler's local copies of that code are gone.What changes
A column's histograms, level histograms and quantiles are checked together. Every one of them goes to flux-stats'
check_edge_countsin a single call (FLUX ADR 0012), and the profile publishes what that call lets through. Before this, each statistic was checked on its own, so a reader could subtract one histogram's published counts from another's, or from a level histogram's group sizes, and rebuild a withheld cell, a zero included. More is withheld now:The summary counts a cell withheld for its size or rounding as
A, and one withheld so that another cannot be recovered asD.A quantile that declares
binsis written asquantile/2. The group gives the declared bin that holds the order statistic and the record counts around it, and no witness. A profile that holds one names format1-1-0. Every other profile keeps1-0-0and the same bytes.The bundle's YAML goes through flux-stats' exact-YAML reader and canonical form. That replaces a copy of the engine's reader that lived in
fluid_build/profiler/bundle.py. The bounds still hold: aliases, merge keys, digits written out, nesting depth, document size, and one read budget shared across a bundle's files. 0.7.0 adds a character budget and a cap on merged mappings. A refusal still names the file and the line, never a value.The
[profile]extra pinsflux-stats[yaml]>=0.7.0,<0.8. 0.7.0 is the first release with these modules, and the range stops at the next minor because 0.x may rename what_stats.pyimports.tests/profiler/test_flux_stats_pin.pykeeps the pin and the install message equal.fluid profile --helpnames the checks.runner.py's two ways of checking a statistic's groups (the edge-count decision, or each statistic's own check) are each a helper now.The CHANGELOG entry sits under Unreleased, after 0.34.0. The released sections, #995's note in 0.33.0 included, are untouched.
Reuse
This is the "depend on it" outcome. The output checks and the YAML reader now come from FLUX's own published library, flux-stats 0.7.0 (Apache-2.0, Python 3.10+, standard library plus PyYAML for the
yamlextra), so forge-cli keeps no second copy. The design behind the checks (ACRO's output checks, tau-ARGUS-style secondary suppression) is in FLUX ADR 0012.Validation
Everything below ran against flux-stats 0.7.0 resolved from pypi.org.
uv pip install --index-url https://pypi.org/simple/pickedflux-stats==0.7.0; both the wheel and the sdist are on the index.scripts/mypy_strict.py, both import-linter configs, and the license headers.tests/profilerplustests/perf/test_startup_budget.py: 501 passed.tests/cli: 3968 passed. Both ran on the final head.test_dotenv_unfiltered_readersfail the same way on main;test_coding_agent_liveneeds a live Claude Code session.fluid profile <contract> --source <slice> --bundle <dir> --out <file>exits 0 with the released wheel, and the document reads back through flux-stats 0.7.0'sload_profile_text.ripwire --quality-delta: the one material regression,_declared_entry's complexity (10 to 17), is fixed by splitting it. The rows left on touched files are this feature's own recent churn and the size of_FluxStatsBackend; the rest are in files this PR does not touch.Reviews
check_edge_countswould have escaped as a traceback;properties[i]andbins[j], never an edge.edge_counts. They are now refused with FLX-5004 before any record is read, by flux-stats' ownepsilon_marginal_histogramrule.Known residuals, not changed here and no worse than before.
fluid profile --helpand the CHANGELOG now state them:support/1lists the levels that hold records;check_edge_countsas a group of 0, although ADR 0012 asks the caller to.These are follow-ups, as is splitting
_FluxStatsBackend, which keeps growing.