Skip to content

refactor(rewrite): express repair and review rules as named findings - #368

Merged
binaryaaron merged 5 commits into
mainfrom
aagonzales/refactor/rewrite-findings
Nov 5, 2026
Merged

binaryaaron merged 5 commits into
mainfrom
aagonzales/refactor/rewrite-findings

Conversation

@binaryaaron

@binaryaaron binaryaaron commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Rewrite's repair and human-review decisions are now named findings, and each risk-tolerance preset lists, as data, which findings send a record to repair and which flag it for review. On the way, a record whose evaluation metrics came back missing or NaN now gets flagged instead of passing review.

Why

The rules were inline arithmetic: determine_repair_needs had two hard-coded triggers and the needs_human_review formula was written twice (staged path, combined graph), each with its own coercions. Adding a condition, as #328 did with replacement_unavailable, meant a new clause in two places with nothing forcing a test or preset entry. The coercions also hid a bug: float(nan) > threshold is False, so a NaN leakage_mass or utility_score passed, and None raised.

What changes

  • Finding is an enum in config/rewrite.py, next to the preset table: rewrite_missing, replacement_unavailable, high_leak, leakage_above_repair, leakage_above_flag, utility_below_flag, unavailable_evidence. The two leakage findings are named after the preset fields they read.
  • engine/rewrite/findings.py (60 lines): present_findings(row, evaluation) -> frozenset[Finding] reads each metric once as float | None, so the type checker enforces the missing-value guard; needs_repair and needs_human_review are the overlap with the preset's repair_on and flag_on. The repair column and both review sites call these; the duplicated formula and its parameter models are deleted.
  • Each _RiskToleranceBundle row spells out repair_on: tuple[RepairFinding, ...] and flag_on. RepairFinding is a Literal of the findings repair can act on (high_leak, leakage_above_repair); listing anything else for repair fails the type check. Thresholds, weights and today's behaviour are unchanged: minimal, low, moderate repair on high_leak or leakage_above_repair, high on leakage_above_repair only, every preset flags the same set. EvaluationCriteria gains read-only repair_on and flag_on; Rewrite, RiskTolerance and the public exports are unchanged; config/rewrite.py imports nothing from the engine.
  • unavailable_evidence (any of leakage_mass, utility_score, any_high_leaked missing or NaN) is in every preset's flag set and never triggers repair; missing metrics no longer raise.
  • docs/concepts/rewrite.md ("Working with flagged records") names the findings and the unavailable-evidence flag. The bundled skill is unaffected.

Not here: the confirmed-leak finding from #358 ("any privacy question answered yes"; a one-line computation plus a preset entry) lands with the preset decision on #358.

About +43 source lines net (+122/-79), including the new module.

Checks

  • ruff format --check, ruff check and ty pass; make test: 1,607 passed.
  • tests/engine/test_findings.py: a table-driven test per finding with threshold cases and NaN, None and missing keys; a test that every finding holds in some case and every preset names only known findings, with unavailable_evidence in each flag set and no repair set; an equivalence grid running the old formulas, copied verbatim from main, against the new predicates for every preset across text, replacement readiness, high-leak, utility and leakage values. Outcomes match exactly, exceptions included, except unavailable-evidence rows, now flagged and never repaired.
  • Commit 3 against commit 2: 134,400 preset-by-row decisions, 0 differences, 0 exceptions.
  • Staged path: RewriteWorkflow.run() with a NaN leakage_mass flags the record and leaves the rest of the frame identical. Combined path: the finalizer flags NaN and None leakage. Both fail without the fix commit.

For reviewers

  • Targeted tests: tests/engine/test_findings.py, plus the unavailable cases in tests/engine/test_rewrite_workflow.py and tests/engine/test_combined_rewrite_workflow.py.
  • One place the two old formulas disagreed: combined checked the rewritten text with is None, staged with isna(). The shared rule uses isna(); they differ only for a float-NaN rewrite, which the combined graph does not produce.
  • Quirk kept because the grid pins it: a NaN any_high_leaked counts as high_leak (bool(nan) is True), None does not.
  • To change a preset, edit repair_on or flag_on in _RISK_TOLERANCE_BUNDLES. To add a condition, add a Finding and one line in present_findings; the coverage test fails until both exist.
  • fix(interface): flag rows whose coverage judge finds missed entities #356 flags missed_entities after evaluate() and could become a finding later; no dependency. feat(rewrite): classify entity labels and enforce sensitivity policy #333 touches the same two workflow files; a trial merge of its head onto this branch has no conflict markers.

🤖 Generated with Claude Code

@binaryaaron
binaryaaron requested a review from a team as a code owner November 3, 2026 17:49
@greptile-apps

greptile-apps Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the latest documentation and test agree with the existing rules.

Summary

The PR puts rewrite repair and review rules in shared named findings and flags missing evaluation metrics for human review.

  • Risk presets now say which rewrite findings trigger repair and review.
  • Records with missing evaluation evidence now get flagged for human review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Evaluated record] --> B[present_findings]
  B --> C[Preset repair_on]
  B --> D[Preset flag_on]
  C --> E[Repair decision]
  D --> F[Human review decision]
Loading

Reviews (4) · Last reviewed commit: "docs(rewrite): name the NaN high-leak ex..." · Reviewed by Greptile

binaryaaron and others added 4 commits November 4, 2026 20:11
Rewrite's repair and human-review decisions were inline arithmetic, and the
needs_human_review formula was duplicated across the staged and combined
paths. Name each condition as a finding (rewrite_missing,
replacement_unavailable, high_leak, protected_leak,
leakage_above_threshold, utility_below_flag, unavailable_evidence) with one
rule per finding, and let each risk-tolerance preset list, as data, which
findings repair and which flag.

Today's presets reproduce today's behaviour: same thresholds, same weights,
high_leak repairs everywhere except `high`, and the flag set is the five
conditions the old formula checked. protected_leak and unavailable_evidence
are computed but not listed by any preset. A grid test checks the new
predicates against the previous formulas, including NaN and None inputs.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
When leakage_mass, utility_score, or any_high_leaked was missing or NaN,
the review comparisons evaluated to False and the record passed without a
flag (or, for None, the run raised). Add unavailable_evidence to every
preset's flag set and read missing metrics as NaN, so a record that cannot
be judged is flagged on both the staged and combined paths.

Repair is unchanged: unavailable evidence flags but never triggers repair.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…t sets

Evaluate each row once against the preset's thresholds into one set of
findings; repair and human review are then the overlap of that set with the
preset's repair_on and flag_on. This replaces the per-call rule dict, the
_Limits carrier and the NaN-coercing _number helper with one dict literal in
present_findings, and metrics read as float | None so the type checker
enforces the missing-value guard.

leakage_above_threshold meant two conditions depending on which tuple listed
it. It is now two findings named after the preset fields they read:
leakage_above_repair (repair_threshold) and leakage_above_flag
(flag_leakage_above).

Finding moves into config/rewrite.py beside the preset table, so the config
module imports nothing from the engine again. Every preset row spells out
both repair_on and flag_on; the shared review set is named once above the
table.

protected_leak is removed. Privacy questions are generated only for the
disposition's protected entities, so the finding meant "any privacy question
answered yes" and was in no preset. It was also the only rule that parsed
payloads. When a preset needs it, compute it beside any_high_leaked and add
one line here.

Behaviour is identical to the previous head: 134,400 preset-by-row repair and
review decisions compared old against new with 0 differences, and the
equivalence grid against the pre-refactor formulas is unchanged apart from
the helper that calls the new API.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
RepairFinding is a Literal of the two findings that repair can act on:
high_leak and leakage_above_repair. It sits directly under the Finding
enum in anonymizer.config.rewrite. Defining a finding now also states
whether repair can act on it. Whether a given preset acts on it is still
decided in the preset table.

repair_on is annotated with RepairFinding, both on _RiskToleranceBundle
and on EvaluationCriteria. The type checker now rejects any preset that
lists in its repair_on a finding that repair cannot act on. For example,
listing unavailable_evidence there is now a type error, not just a test
failure. A new test checks the same rule at runtime through
typing.get_args.

The presets, the call sites, present_findings, the equivalence grid and
the public exports are unchanged, and repair_any_high_leak returns the
same value for every preset.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
@binaryaaron
binaryaaron force-pushed the aagonzales/refactor/rewrite-findings branch from 1c1fe94 to 0ec7aed Compare November 4, 2026 20:13

@andreatnvidia andreatnvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Having the preset table drive both repair and review makes future policy changes easier to follow.

One small clarification: the docs say unavailable evidence doesn’t trigger repair, but any_high_leaked=NaN still produces high_leak through bool(nan) and triggers repair for minimal/low/moderate. Keeping that behavior for compatibility makes sense. Could we mention the exception beside that sentence and add an explicit NaN case to the finding-rule table? The equivalence grid covers it, but a named case would make the intent easier to preserve.

When this moves onto the SDK branch, let’s also carry the unavailable-utility check into its final review projection so it doesn’t overwrite the flag from the combined finalizer.

A NaN any_high_leaked counts as high_leak through bool(nan), so it
triggers repair for minimal/low/moderate. Say so beside the sentence
that unavailable evidence does not trigger repair, and pin it with a
named per-preset test. Behaviour is unchanged.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
@binaryaaron

Copy link
Copy Markdown
Collaborator Author

agent (review-pr): Pushed 7c33e06, which names the NaN exception without changing behaviour. In docs/concepts/rewrite.md, the sentence "Unavailable evidence flags a record but does not trigger repair." is now followed by a note that a NaN any_high_leaked also counts as high_leak and so triggers repair under every preset except high. tests/engine/test_findings.py adds test_nan_any_high_leaked_is_high_leak_and_repairs_except_high, parametrized per preset: it asserts high_leak and unavailable_evidence are both present, repair for minimal/low/moderate, no repair for high, and review in all cases. Carrying the unavailable-utility check into the final review projection is left as a follow-up for the SDK line.

@binaryaaron
binaryaaron merged commit 5b8d0c6 into main Nov 5, 2026
16 checks passed
@binaryaaron
binaryaaron deleted the aagonzales/refactor/rewrite-findings branch November 5, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants