Repository navigation
refactor(rewrite): express repair and review rules as named findings - #368
Conversation
[Medium impact] The PR appears safe to merge; the latest documentation and test agree with the existing rules. SummaryThe PR puts rewrite repair and review rules in shared named findings and flags missing evaluation metrics for human review.
|
Rewrite's repair and human-review decisions were inline arithmetic, and the needs_human_review formula was duplicated across the staged and combined paths. Name each condition as a finding (rewrite_missing, replacement_unavailable, high_leak, protected_leak, leakage_above_threshold, utility_below_flag, unavailable_evidence) with one rule per finding, and let each risk-tolerance preset list, as data, which findings repair and which flag. Today's presets reproduce today's behaviour: same thresholds, same weights, high_leak repairs everywhere except `high`, and the flag set is the five conditions the old formula checked. protected_leak and unavailable_evidence are computed but not listed by any preset. A grid test checks the new predicates against the previous formulas, including NaN and None inputs. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
When leakage_mass, utility_score, or any_high_leaked was missing or NaN, the review comparisons evaluated to False and the record passed without a flag (or, for None, the run raised). Add unavailable_evidence to every preset's flag set and read missing metrics as NaN, so a record that cannot be judged is flagged on both the staged and combined paths. Repair is unchanged: unavailable evidence flags but never triggers repair. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…t sets Evaluate each row once against the preset's thresholds into one set of findings; repair and human review are then the overlap of that set with the preset's repair_on and flag_on. This replaces the per-call rule dict, the _Limits carrier and the NaN-coercing _number helper with one dict literal in present_findings, and metrics read as float | None so the type checker enforces the missing-value guard. leakage_above_threshold meant two conditions depending on which tuple listed it. It is now two findings named after the preset fields they read: leakage_above_repair (repair_threshold) and leakage_above_flag (flag_leakage_above). Finding moves into config/rewrite.py beside the preset table, so the config module imports nothing from the engine again. Every preset row spells out both repair_on and flag_on; the shared review set is named once above the table. protected_leak is removed. Privacy questions are generated only for the disposition's protected entities, so the finding meant "any privacy question answered yes" and was in no preset. It was also the only rule that parsed payloads. When a preset needs it, compute it beside any_high_leaked and add one line here. Behaviour is identical to the previous head: 134,400 preset-by-row repair and review decisions compared old against new with 0 differences, and the equivalence grid against the pre-refactor formulas is unchanged apart from the helper that calls the new API. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
RepairFinding is a Literal of the two findings that repair can act on: high_leak and leakage_above_repair. It sits directly under the Finding enum in anonymizer.config.rewrite. Defining a finding now also states whether repair can act on it. Whether a given preset acts on it is still decided in the preset table. repair_on is annotated with RepairFinding, both on _RiskToleranceBundle and on EvaluationCriteria. The type checker now rejects any preset that lists in its repair_on a finding that repair cannot act on. For example, listing unavailable_evidence there is now a type error, not just a test failure. A new test checks the same rule at runtime through typing.get_args. The presets, the call sites, present_findings, the equivalence grid and the public exports are unchanged, and repair_any_high_leak returns the same value for every preset. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
1c1fe94 to
0ec7aed
Compare
andreatnvidia
left a comment
There was a problem hiding this comment.
Looks good to me. Having the preset table drive both repair and review makes future policy changes easier to follow.
One small clarification: the docs say unavailable evidence doesn’t trigger repair, but any_high_leaked=NaN still produces high_leak through bool(nan) and triggers repair for minimal/low/moderate. Keeping that behavior for compatibility makes sense. Could we mention the exception beside that sentence and add an explicit NaN case to the finding-rule table? The equivalence grid covers it, but a named case would make the intent easier to preserve.
When this moves onto the SDK branch, let’s also carry the unavailable-utility check into its final review projection so it doesn’t overwrite the flag from the combined finalizer.
A NaN any_high_leaked counts as high_leak through bool(nan), so it triggers repair for minimal/low/moderate. Say so beside the sentence that unavailable evidence does not trigger repair, and pin it with a named per-preset test. Behaviour is unchanged. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
|
Rewrite's repair and human-review decisions are now named findings, and each risk-tolerance preset lists, as data, which findings send a record to repair and which flag it for review. On the way, a record whose evaluation metrics came back missing or NaN now gets flagged instead of passing review.
Why
The rules were inline arithmetic:
determine_repair_needshad two hard-coded triggers and theneeds_human_reviewformula was written twice (staged path, combined graph), each with its own coercions. Adding a condition, as #328 did withreplacement_unavailable, meant a new clause in two places with nothing forcing a test or preset entry. The coercions also hid a bug:float(nan) > thresholdis False, so a NaNleakage_massorutility_scorepassed, andNoneraised.What changes
Findingis an enum inconfig/rewrite.py, next to the preset table:rewrite_missing,replacement_unavailable,high_leak,leakage_above_repair,leakage_above_flag,utility_below_flag,unavailable_evidence. The two leakage findings are named after the preset fields they read.engine/rewrite/findings.py(60 lines):present_findings(row, evaluation) -> frozenset[Finding]reads each metric once asfloat | None, so the type checker enforces the missing-value guard;needs_repairandneeds_human_revieware the overlap with the preset'srepair_onandflag_on. The repair column and both review sites call these; the duplicated formula and its parameter models are deleted._RiskToleranceBundlerow spells outrepair_on: tuple[RepairFinding, ...]andflag_on.RepairFindingis aLiteralof the findings repair can act on (high_leak,leakage_above_repair); listing anything else for repair fails the type check. Thresholds, weights and today's behaviour are unchanged:minimal,low,moderaterepair onhigh_leakorleakage_above_repair,highonleakage_above_repaironly, every preset flags the same set.EvaluationCriteriagains read-onlyrepair_onandflag_on;Rewrite,RiskToleranceand the public exports are unchanged;config/rewrite.pyimports nothing from the engine.unavailable_evidence(any ofleakage_mass,utility_score,any_high_leakedmissing or NaN) is in every preset's flag set and never triggers repair; missing metrics no longer raise.docs/concepts/rewrite.md("Working with flagged records") names the findings and the unavailable-evidence flag. The bundled skill is unaffected.Not here: the confirmed-leak finding from #358 ("any privacy question answered yes"; a one-line computation plus a preset entry) lands with the preset decision on #358.
About +43 source lines net (
+122/-79), including the new module.Checks
ruff format --check,ruff checkandtypass;make test: 1,607 passed.tests/engine/test_findings.py: a table-driven test per finding with threshold cases and NaN,Noneand missing keys; a test that every finding holds in some case and every preset names only known findings, withunavailable_evidencein each flag set and no repair set; an equivalence grid running the old formulas, copied verbatim frommain, against the new predicates for every preset across text, replacement readiness, high-leak, utility and leakage values. Outcomes match exactly, exceptions included, except unavailable-evidence rows, now flagged and never repaired.RewriteWorkflow.run()with a NaNleakage_massflags the record and leaves the rest of the frame identical. Combined path: the finalizer flags NaN andNoneleakage. Both fail without the fix commit.For reviewers
tests/engine/test_findings.py, plus theunavailablecases intests/engine/test_rewrite_workflow.pyandtests/engine/test_combined_rewrite_workflow.py.is None, staged withisna(). The shared rule usesisna(); they differ only for a float-NaN rewrite, which the combined graph does not produce.any_high_leakedcounts ashigh_leak(bool(nan)is True),Nonedoes not.repair_onorflag_onin_RISK_TOLERANCE_BUNDLES. To add a condition, add aFindingand one line inpresent_findings; the coverage test fails until both exist.missed_entitiesafterevaluate()and could become a finding later; no dependency. feat(rewrite): classify entity labels and enforce sensitivity policy #333 touches the same two workflow files; a trial merge of its head onto this branch has no conflict markers.🤖 Generated with Claude Code