[binaryaaron]

Rewrite mode now flags a record for human review when its sensitivity disposition leaves a detected entity uncovered, or when a protected entity's original value survives verbatim in the rewritten text. Until now both failures were silent. With a small max_tokens, the disposition can cover only part of a record's entities: the rewrite prompt then handles only the listed ones, the privacy QA does not ask about the rest, and real names reach the output with leakage_mass near 0 and needs_human_review False.

What changes

A new pure module, engine/rewrite/retention.py, makes no model calls and provides two checks.

• disposition_coverage returns the detected entities (keyed on value) and latent entities (keyed on value and label) that have no disposition entry. A non-empty result sets needs_human_review and is recorded in the trace column _disposition_missing_entities. It does not trigger repair, because a repair cannot supply a missing disposition.

• retained_protected returns the disposition entries whose method is not leave_as_is and whose value still appears in the rewritten text. Matching works as follows:

  • both sides are normalized with NFKC, dash and quote folding, whitespace collapse (including U+202F) and casefold;
  • a match must not sit inside a longer word, using (?<!\w)/(?!\w) rather than \b so values like "B.S." and "(555) 010-0142" match;
  • whitespace inside a value matches any whitespace;
  • matches are checked against the original normalized rewrite. A protected match fully contained within a longer kept leave_as_is span is exempt, so a kept "Florida State University" does not count as a retained "Florida". A protected "Florida State University Foundation" extending beyond that kept span is still reported.

  The result is recorded in the trace column _retained_protected_entities. It triggers repair under every risk tolerance preset (max_repair_iterations=0 still disables repair). The repair prompt gains one line per retained entity naming its value and method; for rows with nothing retained, the prompt is byte-identical to before. If the value is still there after repair, the record is flagged for review.

Both Rewrite paths are wired: the staged workflow and Rewrite(use_combined_graph=True). Common-word values such as "History" can match by chance; docs/concepts/rewrite.md ("Working with flagged records") names the two trace columns and this false-positive class.

No public column, signature or default changes. The two new columns are internal and appear only in trace_dataframe. Log lines carry counts only.

Relation to #333

#333 adds a coverage verifier at the disposition step. This change touches none of #333's files and reads only entity_value, entity_label and protection_method_suggestion. Once both land, the two coverage signals overlap, and one should become the single source.

Checks

• make check passes.
• PYTEST_ADDOPTS=--basetemp=/tmp/af353-all make test: 1,655 passed at 72a5dba5.
• tests/engine/test_retention.py: 72 passed. Three new regressions for overlapping names, case/whitespace variants, and multiple occurrences fail before the fix and pass after it.
• Earlier validation also covered tests/engine on pandas 3.0.6.
• New tests in tests/engine/test_retention.py cover, on both paths:
  • a coverage gap flags the record;
  • a retained value triggers repair under each preset and stops once a repair removes it;
  • no repair runs at max_repair_iterations=0, but the record is still flagged;
  • the matching edge cases;
  • the repair prompt text;
  • flags stay on the right record ids when repaired rows are reordered;
  • the retained line reaches the repair call's input when the evaluation returns the column as Arrow lists or nested numpy arrays, as it does after the loop concatenates passing and repaired rows.
• A replay of ten recorded disposition and rewrite results finds 9 of 10 dispositions incomplete. Every real name that survived those rewrites is among the uncovered entities.

🤖 Generated with Claude Code

For reviewers

• Targeted tests: tests/engine/test_retention.py, tests/engine/test_rewrite_workflow.py, tests/engine/test_combined_rewrite_workflow.py.
• The checks are deterministic and make no model calls. Confirm: leave_as_is entries are never flagged; a protected value inside a kept longer value ("Florida" in "Florida State University") is not flagged; a retained value triggers repair under every risk preset; both Rewrite paths (use_combined_graph on and off) carry the two trace columns.
• Matching plan: plans/353/retention-overlaps.md.
• The separate fix for discarded repair iterations follows in fix(rewrite): keep every repair iteration's output on the staged path #354, which must land after this PR.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous overlap issue is fixed and no new blocking issues were found.

Summary

Adds deterministic checks for missing protection decisions and protected values left in rewritten text.

• Rewrite flags records when the protection plan leaves entities out.
• Rewrite retries and flags text that still contains protected values.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Rewritten text] --> B[Check retained protected values]
  B --> C{Values remain?}
  C -->|Yes, repair budget remains| D[Repair text]
  D --> B
  C -->|No, or budget exhausted| E[Check missing protection decisions]
  E --> F[Set human review flag and save evidence]

Loading

_{Reviews (2) · Last reviewed commit: "fix(rewrite): retain protected matches e..." · Reviewed by Greptile}

[asteier2026]

Some of these changes are not necessary with the sensitivity disposition refactor (#333). That PR streamlines the sens disp prompt output so we can handle lots of entities (i.e. require fewer tokens) and there's a check that all entities are covered. As for "retained_protected returns the disposition entries whose method is not leave_as_is and whose value still appears in the rewritten text. ", I"m not sure why we would want this. If the value is still in the rewritten text, it's the job of the Privacy QA to pick that up.

[binaryaaron]

Closing this in favour of #333. Summary of what I checked before deciding, from ten recorded rewrite runs at the default low risk tolerance, with no new model calls.

Coverage. Agreed, #333 makes this half redundant. #333 rebuilds each disposition entry's identity from the supplied inventory by ID and raises when the returned count is short (sensitivity_disposition.py:292-312 at c5d4438). This PR's disposition_coverage returns empty on #333's output; I ran it on a reconstructed disposition to confirm. One difference worth knowing: #333 fails closed (the record is skipped), where this PR emitted the row with needs_human_review. I have left that question on #333.

Retained values. The privacy QA does detect them. In the recorded runs, 5 rows kept a value the disposition had marked for protection (generalize or remove) verbatim in the rewritten text. The QA asked about each one and answered "yes" in 5 of 5. What did not happen is any action:

• 4 of the 5 were medium. Each scored leakage_mass 0.60, so no repair and no review flag.
• One medium "yes" contributes at most 0.6 (config/rewrite.py:30-34).
• Repair needs leakage_mass strictly above the preset threshold, 0.6 / 1.0 / 1.5 / 2.0, and the review flag needs more than 1.0 / 2.0 / 2.5 / 3.0.
• So a single confirmed leak of a value the disposition said to protect is never repaired or flagged under any preset.
• The fifth row was high, scored 1.0, and was repaired and flagged through any_high_leaked, as designed.

That is a threshold-policy question rather than a detection one, so a separate text matcher is the wrong tool for it. I have opened #358 to track it: should a "yes" from the privacy QA on an item the disposition marked for protection trigger repair regardless of leakage_mass? If the answer is yes, the natural place is the QA's own answer, and it can be done on top of #333.

#354 does not use any of this PR's code. It is now rebased onto main and targets it directly.