diff --git a/docs/concepts/choosing-a-strategy.md b/docs/concepts/choosing-a-strategy.md index 75b66fa9..3e6a781c 100644 --- a/docs/concepts/choosing-a-strategy.md +++ b/docs/concepts/choosing-a-strategy.md @@ -140,12 +140,15 @@ Both modes start from the same [detection](detection.md) pipeline. The differenc | Is the goal "scrub the entities and keep everything else"? | ✅ | — | | Is the goal "produce a privacy-safe version of this text that downstream models can train on"? | — | ✅ | | Are there inferable / latent identifiers that aren't explicitly stated (e.g. "during her third round of chemo" → cancer treatment)? | ❌ leaves them | ✅ removes them | -| Additional LLM calls (beyond shared detection) | ~1 (Substitute) or 0 (Redact/Annotate/Hash) | Many (domain → disposition → QA → rewrite → evaluate → repair → judge) | +| Additional LLM calls (beyond shared detection) | Typically 1 per row with entities, plus retries (Substitute); 0 for Redact/Annotate/Hash | Many (domain → disposition → QA → rewrite → evaluate → repair → judge) | | Output text length | ≈ same as input | Often shorter / restructured | | Best for | Structured records, log scrubbing, known-list redaction | Free-text data with implicit identifiers (clinical notes, biographies, depositions, support transcripts) | **Picking between them.** If your data has inferable identifiers that survive entity-only scrubbing (clinical notes, biographies, depositions), Rewrite is the right fit. For structured records, logs, or single-cell PII, Replace is faster and preserves shape. If you're unsure, walk through a few sample rows before deciding. +Substitute's existing whole-response schema retries and its single missing-pair +retry can add calls. See [Substitute validation and retries](replace.md#substitute). + --- ## 4. (Replace) Which strategy @@ -175,7 +178,7 @@ Keep instructions short (one or two sentences). Long instructions compete with t !!! note "Substitute is per-row, not per-dataset" - Within a single row, repeated mentions of the same value get one consistent replacement (entities are grouped by value before the LLM call). **Across rows the LLM has no shared memory** — each row is an independent call, so "Alice" in row 1 and "Alice" in row 47 will likely get different replacements. If you need stable cross-row mappings (e.g. to re-join records by an identifier), use `Hash` instead, or post-process `result.trace_dataframe["_replacement_map"]`. + Within a single row, repeated mentions of the same value get one consistent replacement (entities are grouped by value before the LLM call). **Across rows the LLM has no shared memory** — each row has an independent request context, so "Alice" in row 1 and "Alice" in row 47 will likely get different replacements. If you need stable cross-row mappings (e.g. to re-join records by an identifier), use `Hash` instead, or post-process `result.trace_dataframe["_replacement_map"]`. --- diff --git a/docs/concepts/replace.md b/docs/concepts/replace.md index 29daf31f..4d75e2d9 100644 --- a/docs/concepts/replace.md +++ b/docs/concepts/replace.md @@ -3,7 +3,7 @@ # Replace -Replace mode replaces each [detected entity](detection.md) with an alternative token. Anonymizer provides four strategies. +Replace mode replaces each [detected entity](detection.md) with an alternative token. Anonymizer provides four strategies. --- @@ -22,6 +22,48 @@ Replace mode replaces each [detected entity](detection.md) with an alternative t Replaces entities with LLM-generated synthetic values that are contextually plausible. This is the only replacement strategy that requires an LLM call. +Synthetic values must contain at least one non-whitespace character. Schema +validation rejects empty strings and values made entirely of spaces, tabs, +newlines, or Unicode whitespace. Valid one-character substitutes, such as an +initial or a single-digit age, remain allowed. Valid values are preserved as +generated, including any surrounding whitespace. This shared schema rule also +applies to the replacement maps generated for rewrite mode. + +After schema validation and filtering, Anonymizer checks that every unique +requested `(entity value, label)` pair has a mapping. If mappings are missing, +it retries only the missing pairs once, retaining the accepted substitutions. +A record whose map is complete is returned; a record still missing mappings +after the retry is excluded from successful output and reported in +`failed_records`. Other complete records continue normally. Repeated occurrences +of the same pair need only one mapping, and records without entities need no +replacement-generator call. + +Before discarding a mapping, Substitute can reconcile a copied original that +differs only in no-break spaces (`U+00A0`), thin spaces (`U+2009`), narrow no-break +spaces (`U+202F`), or ASCII spaces. The label must match exactly and the comparison +must identify exactly one pair in the original requested inventory. Exact +mappings take precedence. The mapping's original is restored from that inventory; +the source text, span offsets, and synthetic value stay unchanged. This comparison +does not strip or collapse whitespace or change other characters. Ambiguous or +unmatched entries remain missing and follow the retry/failure flow above. + +This completeness retry applies to schema-valid responses. DataDesigner already +retries schema-invalid responses, validating the record's entire mapping response +on each attempt. Its default is up to five conversation restarts after the initial +attempt, controlled by `max_conversation_restarts` in the `RunConfig` passed as +`Anonymizer(data_designer_run_config=...)`. These schema retries also apply to the +missing-pair generation call. The one completeness retry therefore does not mean +a maximum of two LLM requests. Valid entries from a schema-rejected response are +not retained for a targeted per-entry retry. + +If the completeness retry fails validation after DataDesigner's recovery attempts, +or its workflow fails, the affected records are reported as failed while complete +records are retained. Initial generation retains the existing failure handling: +dropped records appear in `failed_records`, but a failure of the entire initial +workflow can raise `AnonymizerWorkflowError`. Completeness checks exact pairs; +it does not guarantee the semantic quality of a generated substitute or validate +span offsets. + ```python from anonymizer import AnonymizerConfig, Substitute diff --git a/docs/concepts/rewrite.md b/docs/concepts/rewrite.md index c0632218..165ee63e 100644 --- a/docs/concepts/rewrite.md +++ b/docs/concepts/rewrite.md @@ -9,10 +9,14 @@ Instead of replacing individual entities, rewrite mode transforms the entire tex ## How it works -[Detection](detection.md) runs first (same as [Replace mode](replace.md), plus latent entity detection for context-inferable information). This includes identifying signals that may not be explicitly tagged but can be deduced from combinations of details (e.g., location inferred from contextual cues). The text is then classified by domain, and each entity or attribute is assigned a sensitivity disposition based on contextual risk, recognizing that quasi-identifiers can emerge from any aspect of the text. +[Detection](detection.md) runs first (same as [Replace mode](replace.md), plus latent entity detection for context-inferable information). This includes identifying signals that may not be explicitly tagged but can be deduced from combinations of details (e.g., location inferred from contextual cues). The text is then classified by domain, and each entity or attribute is assigned a sensitivity disposition based on contextual risk, recognizing that quasi-identifiers can emerge from any aspect of the text. The text is then rewritten to reduce identifiability, applying targeted transformations that disrupt inference (e.g., weakening or removing linking details) rather than simply rewording content. The rewritten output is evaluated for both quality and privacy leakage using adversarial testing. If thresholds are exceeded, the system automatically refines the rewrite. Any records that failed to meet standards are tagged for human review. +!!! note "Replacement-map validation" + + Rewrite's generated replacement maps share the [Substitute schema rules](replace.md#substitute): synthetic values must contain at least one non-whitespace character, while valid one-character values remain allowed. DataDesigner's existing whole-response schema retries apply to map generation. The single missing-pair retry and Unicode-space reconciliation described for Substitute do not apply to Rewrite; its handling of omitted mappings remains unchanged. + --- ## Key concepts @@ -168,7 +172,7 @@ flagged[["utility_score", "leakage_mass", "any_high_leaked"]].head() - Lower `risk_tolerance` (e.g. `minimal`) to trigger more aggressive repair. - Enable `strict_entity_protection=True` to ensure no detected entity is left unchanged in the output regardless of risk. -**Last resort:** Manually edit or exclude records that resist automated repair — some text is inherently difficult to rewrite without losing utility or leaking identifiers, and requires your judgement as the expert. +**Last resort:** Manually edit or exclude records that resist automated repair — some text is inherently difficult to rewrite without losing utility or leaking identifiers, and requires your judgement as the expert. --- diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 9a6b1f97..ce1d7c96 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -7,7 +7,7 @@ Symptom-first guide to common problems and how to fix them. Each entry says how For a controlled campaign that establishes quality thresholds before testing concurrency, see [Tune quality before throughput](tuning.md). -When something looks wrong, **first confirm the run completed cleanly** (no `failed_records` — these are rows that didn't make it through the pipeline at all, usually a rate-limit / infra issue). Once you know the pipeline ran, **run `preview` and inspect rows with quality issues** (`needs_human_review=True`) — the trace columns it produces are where to start. +When something looks wrong, **first inspect `failed_records`**. Records can be excluded because of model/provider failures, invalid generated responses, or unresolved Substitute mappings. Use each record's `step` and `reason` to distinguish these cases. Once you know the pipeline ran, **run `preview` and inspect rows with quality issues** (`needs_human_review=True`) — the trace columns it produces are where to start. !!! note "This guide is written against the Python API" @@ -21,7 +21,7 @@ Most fixes start with one of these. ### Did the run actually complete cleanly? -Before debugging quality, confirm the run wasn't degraded by infra problems. Anonymizer logs a WARNING with the number of failed records, and every dropped record appears on `result.failed_records` with its step and reason: +Before debugging quality, inspect workflow and substitution-validation failures. Anonymizer logs a WARNING with the number of failed records, and every reported failed record appears on `result.failed_records` with its step and reason: ```python for fr in result.failed_records: @@ -36,14 +36,16 @@ Common patterns and what they mean: | `timed out` | Requests took longer than the alias's timeout, which is 60 seconds when `inference_parameters.timeout` is not set; large `max_tokens` budgets on reasoning models often need longer | Set `inference_parameters.timeout` for that alias in `models.yaml` (the bundled configs use 300). If the reason says local request admission timed out, the request was never sent: lower `max_parallel_requests` instead. See [Models](concepts/models.md) | | `no error was recorded for this row` | The row never completed and left no error of its own, while other rows failed; `most common:` names their most frequent cause | Investigate the named most-common cause first as a run-level hint; it does not establish why this particular row is missing. Only if rows still go missing, check the shutdown error rate in `data_designer_run_config` | | `does not match the source span` | The GLiNER server returned entity offsets that are inconsistent with the text | Restart the self-hosted GLiNER server from a current Anonymizer release (see the [self-hosting guide](concepts/self-hosting-gliner.md)); servers started from older code can return offsets that do not match the text | +| `Replacement map incomplete after one retry` | Substitute still lacks one or more exact requested mappings after its missing-pair retry | See [Incomplete Substitute mappings](#incomplete-substitute-mappings) below; this is separate from schema validation retries | +| `Replacement map retry failed` | The missing-pair generation workflow failed before returning a complete map | Inspect the workflow logs for schema, provider, or request errors; already-complete records are retained | | `Record missing from workflow output` with no other cause, at `step="entity-detection"`, `step="replace-map-generation"` or `step="rewrite-*"` | Often rate limiting (429s) exhausting the retry budget of the model pool | See [Rate limits / dropped rows](#rate-limits-dropped-rows) below | | `Output is missing required tracking column` | Internal; file an issue | None | -If `failed_records` is non-empty, **fix that first.** Strategy/prompt knobs (`risk_tolerance`, `protect`, `gliner_threshold`) won't help: those rows didn't fail because of bad output, they failed because the workflow never produced a usable output for them. +If `failed_records` is non-empty, **investigate those records first**. Provider failures need model/provider configuration changes; incomplete substitution maps need the [Substitute validation and retry checks](concepts/replace.md#substitute). Treat the reported step and reason as the starting point rather than assuming every failure is an infrastructure problem. ### Rate limits / dropped rows -If you have rows failing with `Record missing from workflow output`, the chain is: +For rate-limited rows failing at `step="entity-detection"`, a common chain is: GLiNER candidate set → chunked into validator calls → each call dispatched to a model in the validator pool → `ThrottledModelClient` does AIMD on 429s → if every alias in the pool fails on at least one chunk, the row drops. Fix in this order: @@ -211,7 +213,27 @@ Symptom: every record uses all `max_repair_iterations` and still ends up flagged ### Output rows are missing (`FailedRecord`s) -See [Did the run actually complete cleanly?](#did-the-run-actually-complete-cleanly) and [Rate limits / dropped rows](#rate-limits-dropped-rows). The short version: this is almost always a model/provider issue, not a strategy issue, and the fix is in `models.yaml` / `providers.yaml`. +Start with each failed record's step and reason in [Did the run actually complete cleanly?](#did-the-run-actually-complete-cleanly). Provider errors may require changes to `models.yaml` / `providers.yaml`; incomplete Substitute maps follow the validation and retry behavior below. A workflow-wide error can instead raise `AnonymizerWorkflowError`. + +### Incomplete Substitute mappings + +Substitute compares the schema-valid, filtered map with the unique `(entity value, +label)` pairs supplied before generation. It retains accepted substitutions and +retries only missing pairs once. If the map is still incomplete, the record is +excluded from successful output and reported at `step="replace-map-generation"` +with `Replacement map incomplete after one retry` in its reason. + +Check the missing-label counts in the reason, the requested entities, and the +`replacement_generator` model and `Substitute.instructions`. A copied original +that differs only in a supported Unicode space is reconciled only when it has +one unambiguous match under the same label. Other differences or ambiguous matches +remain missing. See [Substitute](concepts/replace.md#substitute) for the exact rules. + +A response with an absent field, an empty synthetic value, or a whitespace-only +synthetic value fails schema validation instead. DataDesigner applies its existing +whole-response schema retries before the map reaches the completeness check. +Those retries are separate from the single missing-pair retry; this is not a +per-entry recovery mechanism for a schema-rejected response. ### Replacement looks repetitive or unrealistic (Substitute) diff --git a/plans/388/substitution-completeness-retry.md b/plans/388/substitution-completeness-retry.md new file mode 100644 index 00000000..23bbdac5 --- /dev/null +++ b/plans/388/substitution-completeness-retry.md @@ -0,0 +1,79 @@ + + + +# Substitution validation for issue #388 + +For Substitute, compare schema-valid generated mappings with the exact unique +`(entity value, label)` pairs supplied before generation. Retry missing pairs once, +retain already accepted mappings, and fail the affected record if coverage is +still incomplete. Complete records continue and records without entities need no +model call. The user selected this behavior during the investigation of #388. + +## Scope and approach + +Add a validated-map entry point in `engine/replace/llm_replace_workflow.py`, wire +Substitute to it in `replace_runner.py`, and document the successful-output and +failed-record behavior. Keep `generate_map_only()` and rewrite policy unchanged. +Reuse schema validation, filtering, collision handling, and NddAdapter execution. +Snapshot expected pairs before generation; do not infer coverage from entry counts +or model-returned request columns. Track positions so duplicate IDs and reordered +model output retain their original record association. + +Schema validation remains first. This adds one retry for a returned, schema-valid +but incomplete map. DataDesigner's schema-validation retries still apply to each +call. Initial workflow errors retain existing behavior; a wholly failed +completeness retry becomes failures for the affected records while complete +records survive. This work does not introduce permissive transport schemas or +recover invalid entries from a schema-rejected response. + +Reject whitespace-only synthetic values in the shared replacement schema using +`pattern=r"\S"` alongside `min_length=1`. This requires at least one non-whitespace +character in both Pydantic validation and the JSON Schema supplied to DataDesigner. +Keep valid one-character values and preserve surrounding whitespace on valid +strings. The shared rule also applies to rewrite replacement maps; their existing +schema-retry and failure policy stays unchanged. Test the exported schema through +DataDesigner's actual response parser as well as direct validation, mixed batches, +and recovery through configured schema retries. + +## Trade-offs and validation + +A targeted retry avoids regenerating accepted values or making another request +for already complete records. Supply the accepted map as retry context so the +missing replacements can remain consistent. Compare exact pairs after filtering; +an unrelated mapping or duplicate does not fill a gap. Repeated occurrences of +the same pair require a single mapping. + +Before discarding a schema-valid Substitute entry, prefer its exact requested +pair. If it has no exact match, reconcile only U+00A0 (no-break space), U+2009 +(thin space), and U+202F (narrow no-break space) with ASCII space in a comparison +key. The label must match exactly and the normalized key must identify exactly +one pair in the full pre-call inventory. Restore that pair's trusted original +without changing source text, offsets, or the synthetic value. An exact entry +takes precedence over an approximate entry regardless of response order. +Ambiguous matches and other text differences use the existing retry/failure +policy. Do not strip or collapse whitespace, normalize case, or apply broader +Unicode normalization. Keep rewrite's existing exact matching unchanged. + +Test recovery without another model call, exact-entry precedence, ambiguity +across the full inventory on both attempts, exact label matching, unsupported +characters, preservation of offsets/text, and the real preview/batch paths. + +Exercise recovery, terminal incomplete retries, retry workflow drops, mixed +batches, no entities, repeated occurrences, mismatched labels/originals, preview +limits, duplicate IDs and output reordering. Verify preview and batch paths +through the real adapter with a local scripted provider. Failure summaries and +new logs contain counts and labels rather than entity values. Run relevant unit +tests and repository checks. No public API or default configuration changes are +introduced. The bundled skill's guidance is updated; its template imports and API +calls remain unchanged. + +## Rollout + +The three implementation commits, authored and signed off by memadi, are on +`memadi/bugfix/388-substitution-coverage` and published in +[PR #396](https://github.com/NVIDIA-NeMo/Anonymizer/pull/396). They cover missing-map +completeness and one retry, narrow Unicode-whitespace reconciliation, and rejection +of whitespace-only synthetic values. Completeness and reconciliation apply only +to Substitute; the nonblank schema rule also applies to rewrite replacement maps. +Tests and documentation distinguish the existing DataDesigner whole-response +schema retries from the new single missing-pair retry. diff --git a/skills/anonymizer/SKILL.md b/skills/anonymizer/SKILL.md index e8eb0df7..9b8e610c 100644 --- a/skills/anonymizer/SKILL.md +++ b/skills/anonymizer/SKILL.md @@ -31,10 +31,11 @@ regulatory and business context. # Rules - **Always preview before running the full pipeline.** Preview is cheap; a full run can be expensive and slow. -- **If `result.failed_records` is non-empty after preview, fix that *before* tweaking strategy.** Dropped rows are a model/provider/infra problem (rate limits, auth, etc.), not a config problem. Strategy knobs won't help. See `docs/troubleshooting.md` "Did the run actually complete cleanly?" or the [published troubleshooting guide](https://nvidia-nemo.github.io/Anonymizer/dev/troubleshooting/). +- **If `result.failed_records` is non-empty after preview, inspect each failure's `step` and `reason` before quality tuning.** Failed rows can indicate model/provider/infra issues (rate limits, auth, etc.) or invalid or incomplete substitution maps. Treat them as failed records, not successful anonymization. See `docs/troubleshooting.md` "Did the run actually complete cleanly?" or the [published troubleshooting guide](https://nvidia-nemo.github.io/Anonymizer/dev/troubleshooting/). - **Ask the user which mode.** Briefly describe both: Replace detects entities and replaces each in place (faster, cheaper, keeps shape); Rewrite transforms the full text to also remove inferable identifiers (more expensive, may restructure). Use the data shape as a hint — free-text with implicit identifiers (clinical notes, biographies, depositions) leans Rewrite; structured records / log lines lean Replace — but the user picks. - **For cross-record consistency** (same value → same replacement everywhere), use `Hash`, not `Substitute`. `Substitute` is consistent within a row only. - **In Replace mode, default to `Substitute`** if the user hasn't specified a strategy. It's the most general-purpose choice and matches the bulk of production usage. +- **`Substitute` checks mapping completeness before returning a record.** After schema validation and filtering, every unique requested `(entity value, label)` pair needs a mapping. Missing pairs are retried once while accepted mappings are retained; a record still incomplete or whose completeness retry fails is reported in `failed_records`. Unambiguous supported Unicode-space differences can be reconciled under the same exact label. DataDesigner's whole-response schema retries are separate from this single missing-pair retry. The shared replacement schema rejects whitespace-only synthetic values in both modes, but Rewrite does not use Substitute's completeness retry or Unicode-space reconciliation. - **`Annotate` is for inspection, not production.** Its output keeps the original entity text and is not privacy-safe. Use it during iteration to confirm detection is working, then switch. - **Evaluation is opt-in and runs as a separate step** (Replace and Rewrite modes). After `preview()` / `run()`, call `anonymizer.evaluate(result)` to score the output with LLM-as-judge. **Entity coverage always runs** in both modes — it reports detection recall over the judge's unique candidate values (`entity_coverage` + `missed_entities`). On top of that: Replace `Substitute` adds three quality judges (type fidelity, relational consistency, attribute fidelity); Rewrite adds the holistic privacy/quality/style judge. Detection validity is **opt-in** via `EvaluateConfig(compute_detection_validity=True)` (off by default). Evaluation is diagnostic — it scores quality, it does not change the anonymized output. - **Always set `data_summary` on the input**, even briefly. It is the single cheapest quality lever and it improves both detection and rewrite. @@ -190,7 +191,7 @@ def main() -> None: result.trace_dataframe.to_pickle("preview_trace.pkl") print("Saved: preview.parquet (pd.read_parquet), preview_trace.pkl (pd.read_pickle)") - # Failure-first protocol: dropped rows are infra issues, not strategy issues. + # Inspect failure reasons: infrastructure and substitution validation can both fail rows. if result.failed_records: print(f"\n⚠️ {len(result.failed_records)} record(s) failed:") for fr in result.failed_records[:3]: diff --git a/skills/anonymizer/references/interactive.md b/skills/anonymizer/references/interactive.md index 19983509..b588d7fe 100644 --- a/skills/anonymizer/references/interactive.md +++ b/skills/anonymizer/references/interactive.md @@ -31,7 +31,7 @@ Iterative design with the user. Do not disengage from the loop until the user sa 5. **Build** — Write the Python script to the current directory using the Output Template in `SKILL.md`. Name the file after the dataset (e.g. `anonymize_clinical_notes.py`). 6. **Preview** — Run `python