Repository navigation
fix(rewrite): keep every repair iteration's output on the staged path - #354
Conversation
The staged evaluate-repair loop joined each repair result onto the failing rows without overwrite. From the second repair on, the rows already carried the first repair's `_rewritten_text__next`, so the new repair output was dropped: the row kept its first repair's text, the re-evaluation scored that same text again, and `_repair_iterations` still went up. Setting `max_repair_iterations` above 1 only added model calls. Join the repair result with `overwrite=True`, excluding the repair seed columns as the re-evaluation join already does, so the adapter's serialised copies of the seed never replace the frame's values. The repair step's own outputs, `_leaked_privacy_items` and `_rewritten_text__next`, now land on every iteration. Behaviour with a single repair is unchanged; the combined path keeps each iteration in its own columns and was not affected. Tests use adapter-shaped stubs with distinct text per repair, a run through the real adapter with deterministic columns, and a staged-vs-combined comparison that takes the combined result from the real graph finalizer. 🤖 Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryThe staged rewrite path now keeps each repair’s latest output while preserving the columns passed into the repair step.
No actionable issues were found. Tests were inspected but not run during this review. Reviews (4) · Last reviewed commit: "fix(rewrite): keep every repair iteratio..." · Reviewed by Greptile |
Published as The updated #353 prerequisite was merged without rewriting either branch, and this PR now targets its branch. Merge #353 first. The remaining diff is still the one-line production fix plus its tests. Validation: 127 focused rewrite/combined/retention tests passed; full |
|
I would like to approve the changes in this PR, but it's stacked on PR 353 which I don't think we want. |
c1d6867 to
f228263
Compare
andreatnvidia
left a comment
There was a problem hiding this comment.
Looks good to me. Overwriting the repair outputs while preserving the seed columns fixes the repeated-repair behavior and follows the existing re-evaluation pattern.
|
One follow-up for the graph line: #319 and #365 still call |
In Rewrite mode, every repair after the first was discarded. The evaluate-repair loop joined each repair result onto the failing rows without
overwrite; from the second repair on, the rows already carried_rewritten_text__next, so the join kept the first repair's text. Re-evaluation scored that same text again,_repair_iterationsstill went up, andmax_repair_iterationsabove 1 (the default is 3) only added model calls.The fix
One line in
src/anonymizer/engine/rewrite/rewrite_workflow.py, mirroring the re-evaluation join below it:overwrite=Truelets the repair step's outputs (_leaked_privacy_items,_rewritten_text__next) replace the previous iteration's.seed_cols=repair_seed_colsprotects the columns the step only read, which the adapter echoes back serialised (_privacy_qawould come back as a JSON string): the record id, original text, current rewrite, rewrite baseline,_privacy_qa,_privacy_qa_reanswer,leakage_mass,any_high_leaked,utility_score.Behaviour
One repair: no change. More than one: each repair sees the previous repair's text and re-evaluation scores the new text; a row that passes after its second repair stops with
_repair_iterations == 2. For rows repaired more than once, the trace columns_leaked_privacy_itemsand_rewritten_text__nextshow the last iteration; column set and dtypes are unchanged. The combined path (Rewrite(use_combined_graph=True)) is unaffected: it writes each iteration to its own columns.Checks
make checkpasses. Full suite: 1,587 passed atf228263bonmain(f294dc1e). Earlier validation also coveredtests/enginewith pandas 3.test_repair_loop_keeps_each_iterations_repaired_text: three repairs with distinct text, row never passes; final text is the third repair's, counter 3, each later step receives the previous repair's text. Without the fix the final text is the first repair's.test_repair_join_keeps_seed_columns:_privacy_qaseeded as a JSON string stays a dict. Passes before and after; fails ifseed_colsis dropped.test_repair_loop_mixed_rows_keep_their_last_repair: rows passing after 0, 1, 2 or never; final text, counter, rows sent to each repair and output order all checked.test_repair_loop_stops_after_successful_repair: passes after the second of three allowed repairs.test_repair_loop_through_real_adapter_keeps_each_repair: through the realNddAdapterwith deterministic custom columns, no model requests.test_combined_and_legacy_paths_return_equivalent_repaired_result(strengthened): two distinct repairs; the combined result comes from the real graph finalizer through the real adapter.main(the seed-guard test is meant to pass on both). Removingoverwrite=Truefails the same five; removingseed_colsfails the seed-guard and real-adapter tests.Previously stacked on #353; it does not depend on that code and now targets
maindirectly.🤖 Generated with Claude Code