[binaryaaron]

Related Issue

Addresses #358. This PR originally addressed the gap with a separate confirmed_leak finding; it now changes the risk_tolerance thresholds instead, so sensitivity and confidence keep their role in leakage mass.

Scope: the issue asked for repair regardless of leakage mass; this change does not do that. A confident medium-sensitivity leak is now repaired on minimal and low, never on moderate or high, and one that survives repair is flagged on minimal only.

Plan Document

No plan required: the change edits four threshold values in one table, plus the tests and docs that quote them.

Summary

Lowers the repair and review thresholds of the minimal, low and moderate risk_tolerance presets so that a single confident medium-sensitivity leak is repaired on the stricter presets.

Why it matters: leakage-based repair needs a leakage mass strictly above the preset's repair threshold. One medium-sensitivity entity leaked at full confidence has mass 0.6 (weight 0.6 x confidence 1.0). That is not above the old minimal threshold of 0.6, and it is below the old low, moderate and high thresholds, so no preset repaired it and none flagged it either (the old minimal review threshold was 1.0).

New values in _RISK_TOLERANCE_BUNDLES (src/anonymizer/config/rewrite.py), repair threshold / flag leakage above:

Preset	Before	After
minimal	0.6 / 1.0	0.4 / 0.5
low	1.0 / 2.0	0.5 / 2.0
moderate	1.5 / 2.5	1.0 / 2.5
high	2.0 / 3.0	2.0 / 3.0 (unchanged)

flag_utility_below, repair_on and flag_on are unchanged. Repair thresholds remain strictly ascending (minimal < low < moderate < high), and a test now asserts this. The minimal repair value is a one-line edit in the preset table if the pending confirmation below goes another way.

Resulting behaviour for one confident medium-sensitivity leak (mass 0.6): repaired on minimal and low, not repaired on moderate or high; if it survives repair, flagged for human review on minimal only. Two such leaks (mass 1.2) also repair on moderate, but not on high. A faint low-sensitivity leak (weight 0.3 x confidence 0.05 = 0.015) triggers neither repair nor review on any preset. The sensitivity x confidence weighting of leakage mass and the meaning of risk_tolerance are unchanged; only where each preset draws the line moves.

User-visible effect: more repair calls on minimal, on the default low preset and on moderate, and more records with needs_human_review=True on minimal.

Pending confirmation

The minimal repair threshold is set to 0.4, pending maintainer confirmation. Changing it is a one-value edit in the preset table (_RISK_TOLERANCE_BUNDLES). Alternatives:

• 0.25: also repairs judge-inferred leaks at confidence of about 0.42 and above (medium weight 0.6), at the cost of more repair calls.
• 0.5: equals the low repair threshold, so low would also have to be raised to keep the order strict.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Refactoring
• CI, release, or contributor workflow update

Default behaviour of risk_tolerance presets changes (see User-visible effect above); no public signatures change.

Contributor Checklist

• PR title follows Conventional Commits: fix(rewrite): repair confident medium leaks on the stricter risk_tolerance presets
• Related issue is linked (Rewrite: a confirmed medium leak of a protected value is never repaired or flagged under any preset #358)
• No plan document needed; reason documented above
• Public API impact checked; skills/anonymizer/SKILL.md needs no update (its template only passes risk_tolerance="low" and quotes no threshold values)
• No real PII added to tests, docs, notebooks, fixtures, or artifacts
• No API keys, service tokens, private keys, credentials, or real endpoint secrets added

Validation

Unit and static checks:

• Commands run: ruff format --check and ruff check (tools/codestyle/format.sh --check, ruff_check.sh), tools/codestyle/typecheck.sh (all dependency groups), the full pytest suite, mkdocs build --strict (exit code 0).
• Skipped checks or known failures: full suite 1788 passed, 2 skipped, 1 failed. The failure is test_wandb_staging_allows_repeated_basename_in_foreign_intermediate_directory, which is unrelated to this change.
• New tests in tests/engine/test_findings.py use the real present_findings, needs_repair, needs_human_review and compute_leakage_mass: a confident medium leak repairs on minimal and low only; two confident medium leaks (mass 1.2) repair on moderate but not high, and are flagged on neither; a low-sensitivity leak at confidence 0.05 triggers neither repair nor review on any preset; thresholds are strictly ordered. Exact-value assertions in tests/config/test_rewrite.py are updated.

Live run (one dataset, one judge model, minimal preset only; there is no live run for low, the default, or moderate): the public docs/data/NVIDIA_synthetic_biographies.csv, 23 of 25 rows processed (2 dropped by a GLiNER span-mismatch check), a local gpt-oss-120b judge, three repair iterations. "Leaking" below means a record whose final leakage mass is above 0.

• Main thresholds, one run: 4 of 23 records had a confirmed medium-sensitivity leak (leakage mass 0.48 to 0.6) that was neither repaired nor flagged, and ended leaking. 1 of 23 was flagged for review.
• Two runs with changed thresholds used minimal at 0.25 / 0.5 (repair threshold / flag leakage above), not the 0.4 / 0.5 proposed here. Both repaired those records and ended with 0 leaking records; 1 of 23 was flagged for review in each, the same as on main.
• Model requests: 323 and 323 (the two changed-threshold runs) versus 303 (the single main run), about 7% more.
• Mean utility of the repaired records, before to after repair: 0.928 to 0.803 (first changed-threshold run), 0.912 to 0.779 (second), and 0.946 to 0.872 for the main run, whose repairs were for high-sensitivity leaks only.
• Every record the changed-threshold runs repaired at first evaluation for a medium-only leak had leakage mass of 0.525 or above (0.525, 0.54, 0.54, 0.6, 0.6, 0.6). No evaluation in either run, first or later, had leakage mass between 0.25 and 0.4: records re-repaired in later iterations had 0.57 or more or a high-sensitivity leak, so 0.4 would have made the same repair decisions on these evaluations. 0.4 itself was not run, and judge answers vary between runs, so the "0 leaking records" result is not measured at 0.4.

Run-to-run variation is a few records, and this is a single dataset; the figures show direction, not a measured rate.

Documentation and Artifacts

• Docs updated: docs/concepts/rewrite.md (risk-tolerance table, plus one sentence on what a single confident medium-sensitivity leak does under each preset), docs/troubleshooting.md (threshold table), and the RiskTolerance docstring. The repo keeps no changelog file.
• mkdocs build --strict run locally (the equivalent of make docs-build, with output written outside the repo); the site built, with only an INFO message about a missing #reference-implementation anchor in tuning.md that this change does not touch.
• No tutorial sources changed; notebooks were not regenerated.
• Live-run evidence is listed above; no e2e or benchmark files changed.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The current threshold changes appear safe to merge.

Summary

This PR lowers leakage thresholds rather than adding the broader confirmed_leak behavior described in its title and description.

• Rewrite presets now repair and flag records at lower leakage levels.

No actionable issues were found in the current changes. Tests were not run during this review.

_{Reviews (3) · Last reviewed commit: "fix(rewrite): repair confident medium le..." · Reviewed by Greptile}

[asteier2026]

I feel like this undermines both our "risk_tolerance" parameter, and the way leakage mass utilizes both sensitivity and confidence. But it does seem wrong that no setting will repair a high confident medium sensitivity entity miss. My preference would be to adjust the thresholds for minimal, low, moderate settings. I just don't think we want a blanket always repair when confidence scores are low.

[lipikaramaswamy]

I am also curious how this interacts with #333, which should enable better sensitivity assignments. I guess it doesn't change a poor rewrite. I agree that the whole point of having minimal, low, moderate, high options for risk tolerance is to somewhat control the privacy utility tradeoff. What would you recommend changing thresholds to @asteier2026?

[binaryaaron]

@lipikaramaswamy @asteier2026 - i'm running a few tests internally and will report back on updated thresholds. sorry, i think this one was premature.

[binaryaaron]

@asteier2026 @lipikaramaswamy: changed this to update the thresholds instead.

Preset	Repair / flag-leakage, now	Proposed
minimal	0.6 / 1.0	0.4 / 0.5
low	1.0 / 2.0	0.5 / 2.0
moderate	1.5 / 2.5	1.0 / 2.5
high	2.0 / 3.0	unchanged

One confident medium leak now repairs on minimal above confidence 0.67 and on low above 0.83. moderate needs two.

I tested these locally on docs/data/NVIDIA_synthetic_biographies.csv, with minimal at 0.25 / 0.5 (not 0.4). Main left 4 records with a medium leak neither repaired nor flagged; two runs with the changed thresholds left none and flagged 1 of 23, as main did, with

• 20 more requests compared to main (323 vs 303)
• with slightly lower utility on repaired - (0.928 to 0.803, 0.912 to 0.779)

Question

wdyt about 0.4 for minimal? 0.25 would also repair inferred leaks from confidence about 0.42

#333. - i can help revalidate the thresholds after we land that? wdyt?

[asteier2026]

I'd be comfortable with those thresholds. But I also like:
min: .25/.6
low: .5/1
mod: 1/2
high: 2/3

[binaryaaron]

agent (review-pr): Pausing the threshold choice here until #333 lands. #333 goes first, and it changes the leakage-mass distribution these values were tuned on. In two comparison runs on the 25-row public biographies CSV, the lowest first-evaluation mass on #333's tree was 1.2 in one run and 0.96 in the other. Both proposals below set the minimal, low and moderate repair thresholds at 1.0 or lower, so under either one each of those three presets would repair 24 or 25 of the 25 records, and they would barely differ. Once #333 is on main, the thresholds will be re-derived on top of it, and this PR will be rebased and re-scoped to presets that still separate.

For the record, the evidence gathered on main's code so far:

• The three sets compared were A, main (0.6/1.0, 1.0/2.0, 1.5/2.5, 2.0/3.0, repair / flag for minimal / low / moderate / high); B, this PR's head (0.4/0.5, 0.5/2.0, 1.0/2.5, 2.0/3.0); and C, the values @asteier2026 suggested (0.25/0.6, 0.5/1.0, 1.0/2.0, 2.0/3.0).
• In a live run on main, B and C repaired the same records on every preset. They differ only for a first-evaluation mass between 0.25 and 0.4, which no public run reached.
• With C's minimal flag threshold of 0.6, a record still above its 0.25 repair threshold after every repair round can end unflagged. One record did, at 0.51. If C's shape is kept after the re-derivation, minimal's flag threshold should not exceed what the repair loop can leave behind.

On #333's tree, the existing weighted_leakage_rate column still separated the presets where fixed mass did not. It is worth considering as the quantity the presets threshold on.

Edited: corrected the lowest first-evaluation mass (1.2 in one run, 0.96 in the other) and the repair count under either proposal (24 or 25 of 25, not every record). The conclusion is unchanged.

[binaryaaron]

agent (review-pr): Edited the reply above to correct two figures. The conclusion and the pause on the threshold choice are unchanged.

• The lowest first-evaluation mass on feat(rewrite): classify entity labels and enforce sensitivity policy #333's tree was 1.2 in one run and 0.96 in a second, not 1.2 throughout.
• Under either proposal, minimal, low and moderate would each repair 24 or 25 of the 25 records, not every record.