Repository navigation
refactor(detection): establish deterministic entity candidate lifecycle - #305
lipikaramaswamy wants to merge 2 commits into
Conversation
lipikaramaswamy
changed the title
lipikaramaswamy
changed the title
binaryaaron
left a comment
binaryaaron
left a comment
There was a problem hiding this comment.
agent (
review-pr): Plan feedback — shared detection semantics and composable backends.
The shared lifecycle is the right foundation for #265 and the eventual #302 integration. Following a council review, I suggest adding a narrow internal entity-detection backend contract to this plan, with GLiNER first and regex added when #265 is rebuilt. The four inline comments identify contracts to clarify before implementation; this is a review of the plan at fd1fea9b, not an approval of production code.
The intended sequential composition is:
source -> regex -> regex observations --------------------+
| |
+-> optional GLiNER context |
source -----------------> GLiNER -> new observations -----+
|
collect
|
shared admission / derivation / reconciliation
Without the regex-to-GLiNER data binding, the two backends can execute independently. Ordering and data consumption should be explicit: running GLiNER after regex does not automatically mean it consumes regex output.
Ownership
- Backends produce source-bound observations and distinguish successful empty output from failure.
- The shared detection lifecycle owns candidate admission, label eligibility, accepted provenance, propagation, and overlap reconciliation.
- Graph bindings own execution dependencies, request associations, retries, resource limits, and artifact provenance. Qualification of the final transformed output remains separate from detection acceptance.
Identity and reuse
Assign immutable, source-scoped observation references before publishing a batch for downstream reuse. Collection preserves them; short validator tokens such as c1 can be request-local aliases. Do not allocate identity from completion order or regenerate it during merging.
GLiNER can produce new claims while using regex context, and those claims follow its own admission route. Forwarded regex observations retain their original identity and restrictions; passing them through GLiNER cannot create another supporting origin. Derived claims retain explicit parent references. Graph-level dependency records influence, while entity-level provenance determines authority. The simplest initial collection contract retains the original regex batch and adds GLiNER's newly produced observations.
Scope
A private _EntityDetectionBackend ABC is reasonable. Introduce the observation contract and a narrow GLiNER boundary while preserving the existing DataDesigner execution/export path. Add the regex implementation with #265 and graph execution bindings during #302 integration. An async detect() method alone does not express the effect, batching, cancellation, and resource contracts the SDK needs. Keep graph imports out of the domain types.
Define a concrete supported reuse behavior before standardizing a generic hints interface. Supplying context, selecting labels, masking text, and skipping detector work have different coverage consequences. Unsupported context should not silently be ignored. Keep augmentation's invocation contract separate initially while allowing it to produce compatible observations and use the shared lifecycle.
Validation
Use shared semantic fixtures for the pure helpers, serialized DataDesigner execution, and eventual graph adapter. Include the four inline scenarios, sequential reuse without duplicated authority, preservation of both source batches, and serialization round trips. Pin benchmark inputs, configuration, baseline, and acceptance criteria, including coverage changes from fallback and seed-over-augmentation precedence.
The excluded-rule export finding remains a separate #265 item. Also align this plan's older allow_model_detection / validate_with_llm references with #265's current detect_additional_matches / validate_matches_with_llm names.
|
Addressed the plan review in 1bce7b7, with terminology clarified in 5f38e04. The revised plan now defines:
I also updated #304 so unusable label proposals receive exact-span fallback, while usable but excluded or disallowed proposals are rejected and traced separately. |
|
When we have overlapping spans where one is contained in another, I definitely see that taking the longer span makes sense. It's just not clear to me what the right thing to do is when the spans overlap. Ideally, we would use sensitivity but we won't have that information until after the sensitivity disposition. I'm thinking this situation won't happen often and it's probably not worth worrying about. Taking the longest span is a good default. |
|
It's not clear to me what happens when we have "May called yesterday. The appointment is in May". Could we end up tagging both "May"s as names? |
binaryaaron
left a comment
binaryaaron
left a comment
There was a problem hiding this comment.
The source/view identity and execution-scoped admission changes address the main concerns Andrea raised. A few implementation details will matter when these components are used in #302:
- Collect all observations before resolving overlaps. With A=
[0,4), B=[3,9), and C=[8,15), longest-first selection across all three retains A and C. Resolving A/B first and then adding C retains only C. Add a test showing that collecting results from graph branches does not discard candidates before the final reconciliation step. - Preserve each origin’s permissions after merging equivalent candidates. If a regex observation prohibits propagation and an accepted augmentation observation permits it, only the augmentation observation supports protecting a new occurrence. A combined propagation flag and a list of origin IDs would lose that distinction.
- Convert offsets to the original source coordinates exactly once. The source section says backends do this before publishing observations, while Step 2 also describes converting offsets. Specify which step performs the conversion and which checks the result.
- Keep detector behavior separate from workflow execution. The private backend interface should describe detector inputs and results. NDD and future graph adapters should manage transport, retries, scheduling, and result publication. Validation and merging should remain shared operations outside the backend classes.
#302 can track which inputs and artifact versions an operation used. The detection adapter will still need to check serialized observations and associate each validation result with the source text, context, and policy used to produce it. Reusing an observation should preserve its detector identity, but should not automatically reuse an earlier validation result. Accepting an entity also does not establish that the graph’s final output qualifies for Protected release.
These details can be specified and tested in #305 without implementing graph integration or validation under multiple policies yet.
|
Say we had: |
Yes, this can happen today. This behavior has existed since the original replace-mode implementation: GLiNER finds one occurrence → validator confirms the value is sensitive → protect every identical occurrence. The motivation was reasonable at the time
So we have been making the semantic assumption that if two spans have the same text, they have the same label and meaning. However, this assumes that equal text has equal meaning. That is not true for your example, or something even more complex like I no longer think global blind propagation should be the standard final stage. A better target workflow would be: The augmenter currently returns: Because that result has no occurrence anchor, the engine materializes it at every matching occurrence. The proposed response would be:
We should benchmark the recall effect of removing blind propagation. For highly stable identifiers, exact expansion may still be useful as an explicit policy, but perhaps relying on regex / gliner is sufficient. For contextual entities such as names, locations, and organizations, additional occurrences should receive independent decisions. I think #305 should establish occurrence-specific authority now; otherwise the new lifecycle risks formalizing the existing assumption that accepting one occurrence authorizes every equal string. |
|
Regarding the "May" example. Your suggestions may work. My gut feeling is it will happen way more often that gliner/augment will miss repeated instances of the same value/label than we will have ambiguous cases like "May". |
Good catch @asteier2026. We probably should not use normalization or response ordering to choose between duplicate entries. How about this: we discard the invalid response entirely, and retry the same validation request. With a validator pool, the retry proceeds to the next alias in pool order. With a single validator alias, we will allow one bounded same-alias semantic retry. If all attempts return invalid responses, the request follows the existing terminal FailedRecord path. This is distinct from an omitted alias or a single unusable entry in an otherwise valid response. Those cases retain exact-span fallback protection without propagation rather than retrying the request. |
|
Follow-up on @asteier2026's concern that GLiNER or augmentation may miss repeated occurrences more often than equal-text ambiguity occurs: I audited 12 completed trace artifacts covering 986 full-run records across Gemma, GPT-OSS, and Qwen on RAT-Bench, Nemotron-PII, and OpenPII.
This does not prove that repeated-mention recall is unimportant. It does show that unconditional propagation is doing much more than recovering missed repeated names and identifiers, and that the assumption "equal text means equal entity meaning" creates substantial noise. The revised plan therefore removes blind |
|
The implementation plan has been substantially revised in 9854bff, and the settled review threads have been updated and resolved. One experiment remains explicit in the plan: compare seed-tagged versus canonical plain-text augmenter input on the same RAT-Bench, Nemotron-PII, and OpenPII samples, especially duplicate rediscovery versus genuinely new recovery. If there are no additional comments by end of day Wednesday, October 21, I will start implementation. |
9854bff to
97c0ae8
Compare
Implementation ready for review#305 changes how detections move through the pipeline: each detected mention is tracked separately, each validation decision is applied once, and all accepted findings go through one consistent rule for resolving overlaps. It also stops automatically tagging every identical occurrence without checking its context. This avoids treating a person named “May” and the month “May” as the same thing. The tradeoff is that repeated sensitive mentions can remain unprotected if both the detector and augmenter miss them. What we testedWe processed the same 600 records with the baseline and #305 versions, using three models: 3,600 processing attempts. Models: GPT-6 Luna, Nemotron 3.5 Super VL and Gemma 4 26B A4B. GPT-6 Luna and Nemotron 3.5 Super VL ran through NVIDIA’s inference endpoint. Gemma 4 26B A4B ran through OpenRouter. Gemma primarily used PrimeIntellect’s BF16 endpoint; failed #267-prompt attempts were retried on CoreWeave’s BF16 endpoint. Provider fallback was disabled.
The Nemotron/OpenPII source repository requires access. Both versions used identical saved GLiNER detections, input records and date guidance. We tested annotation only—not substitute generation, rewriting or re-identification. Precision and recallFor Nemotron PII, OpenPII and Privacy-Bench, we compared the letters and digits covered by detections against the supplied annotations:
We counted only entity types covered by each dataset and combined the character counts across the three datasets. These measurements do not require an exact entity boundary or label match, and they are not a substitute for manually judging every finding.
Only records successfully returned by both versions are compared. RAT-Bench is excluded from this table because our reference spans are derived by matching values in its identifier metadata. They are not a complete list of every sensitive mention in the text. Across all six workloads:
Checking repeated mentionsWe inspected changed records and repeated a smaller test three times on examples with known misses, alongside examples both versions previously handled correctly. Some repeated-name misses occurred consistently. For Luna and Super, the augmenter either omitted the missing mention or found the same text elsewhere. The findings were not accepted and then accidentally removed later. Gemma also missed some suggestions because their supplied context did not match the source text exactly. This confirms that removing automatic copying exposes a real recall limitation. Follow-up: the plain-text prompt in #267We separately tested #267’s approach on #305: give the augmenter plain text, remove the list of existing detections, and remove the instruction not to repeat already-tagged entities. Everything after the model response stayed unchanged.
Each annotated occurrence checked in three repeated runs contributes three checks. These are selected difficult examples, so the numbers do not estimate recall across the full datasets. The improvement took more time. Average augmentation time, including retries, increased from 19→28 seconds for Luna and 19→32 seconds for Super. All 34 Luna/Super attempts returned. Gemma returned 48/51; three responses still failed the required output format. Failed Gemma attempts were retried on another provider, while successful results were retained. The comparison reused saved current-prompt results rather than rerunning both prompts together. The Gemma regression was one organization mention omitted in all three repeats—not a finding removed by later processing. #267 remains a separate follow-up. Its prompt recovered substantially more of the selected missed mentions, but adds time and still has a regression to investigate. The published #305 version passes CI and DCO. We’re opening it for review with these benefits and limitations documented—not claiming that every detection-quality issue is solved. |
 [High impact] No new code issues block merging, but the stated DataDesigner dependency gate still applies. SummaryThis PR gives detections a shared path from source-bound observations through validation, admission, augmentation, and final overlap selection.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| AugmentationConfig( | ||
| name=COL_AUGMENTED_ENTITIES, |
There was a problem hiding this comment.
Replacing LLMStructuredColumnConfig with AugmentationConfig removes the augmenter’s message traces when dd_trace is enabled. The adapter traces built-in LLM columns or CustomColumnConfig model calls, but this new config matches neither. Users lose the prompts and responses needed to explain missed suggestions and failed corrections. Register the plugin’s model calls with the existing trace path.
There was a problem hiding this comment.
Addressed in e0dd247 (the rebased plugin-tracing fix), retained through the main rebase and current head 399ede8. Model aliases are now discovered through column.get_model_aliases(), including plugin columns, and their facade calls enter the existing private model-call trace path rather than requiring CustomColumnConfig.
The augmentation-plugin trace regression covers trace disabled/last-message/all-messages, successful extraction, grounding correction, transport failure, and parse failure. It checks column association, async trace flags, raw-message isolation from sanitized measurements/logs, and restoration of patched facade methods. Validation and augmentation plugins now use the async-only execution path inherited from main; no removed sync bridge is restored. All 1,844 tests and make check passed at 399ede8.
Build regex detection on PR #305. Preserve bounded matching, local validators, per-rule admission routes and regex-only labels while using the shared observation admission and reconciliation path. Keep rule claims independent until admission and require exact accepted regex evidence for model-derived regex-only labels. Do not restore value propagation or replay validator decisions during finalization. Signed-off-by: lipikaramaswamy <lramaswamy@nvidia.com>
3ab3eb6 to
b643216
Compare
|
@binaryaaron — addressed the offset-mismatch guard in C3 in The GLiNER backend now raises This is a defensive contract restoration, not a claim that our current GLiNER endpoint has a demonstrated offset bug. The inspected workflow preserves exact detector input, and the bundled local GLiNER2 path validates chunk spans before remapping them. If an inconsistent response nevertheless reaches this boundary, it must fail rather than produce an apparently successful result with a silently omitted detection. Replaced the old skip-on-mismatch test with rejection cases for both an incorrect reported value and the synthetic emoji/UTF-16-style offset example, including a valid observation before the invalid one. Existing canonical-source and source-view mapping coverage remains green. Verification: 84 focused backend/source tests passed; 1,834 full-suite tests passed; The related malformed-payload/empty-result behavior is unchanged in this narrowly scoped fix. We will correct the broader “distinguishes no findings from failure” description when doing C6, as discussed. C4 remains separate and is not claimed fixed here. |
|
@binaryaaron — addressed C4 in pushed commit
Regression coverage exercises the actual response parser with an extra top-level We also checked one saved Gemma failure: its original response parses with tolerant handling. A separate one-request Gemma/CoreWeave BF16 probe requested strict provider-native JSON-schema output with closed objects, required parameter support and no provider fallback. It returned a schema-valid response in 17.6 seconds; 41 of 43 suggestions grounded, while two still failed non-verbatim-context checks. This is a narrow feasibility result, not proof of semantic correctness, complete record recovery or universal endpoint enforcement. Native schema enforcement is not enabled globally by this fix. Detection-scoped validation/augmentation support, including corrections/retries, provider compatibility, explicit unsupported-mode behavior and tracing, is tracked separately in #372. The current tolerance change remains useful for the existing prompt-and-parser path. |
|
@binaryaaron — response to C1: the intended choice is (b), occurrence-specific detection plus the plain-text augmenter, not restoration of automatic propagation/name splitting. #267 now targets this PR's branch, Division of responsibility
You are right that #305 by itself still has the old prompt restriction. We are not claiming that restriction is compatible with relying on the augmenter for missed mentions; #267 is the explicit stacked change addressing it. Likewise, if GLiNER observes only the first Case-sensitive propagation would avoid the Measured evidence and limitationsWe tested a prompt-only adaptation of the original #267 onto #305 at
Checks include repeated runs, not counts of unique records. These are selected missed-mention checks, not a comprehensive repeated-occurrence or homograph benchmark. All 34 Luna/Super attempts returned; Gemma returned 48/51 after failure-only cross-provider recovery, with three schema failures. Its control regression was an organization omitted by the model in all three repeats, not a finding suppressed by reconciliation. This is a small failure-enriched cohort using saved current-prompt outputs, not a fresh counterbalanced comparison; Gemma's provider change is another confound. It supports recovery of selected misses but does not establish population precision/recall, guarantee all repeats/name parts are protected, or fully satisfy the broader repeated-mention/homograph evaluation proposed in the plan. The fresh paired evaluation remains outstanding and is recorded in #267's plan. We will update #305's stale PR description and notebook/concept documentation in C6, after the behavioral decisions/fixes, so they explicitly describe removal of propagation/name splitting and the #267 stack. This response records the intended policy and dependency; it does not claim the prompt change is already merged into #305 or that the remaining coverage trade-off has disappeared. |
|
@binaryaaron — addressed C8 in pushed tests-only commit The ranking tests now include same-start narrow/broad spans and a later-start, longer partially overlapping span, in addition to the original earlier-start case. A separate To check that these discriminate the comparator, we removed only the negative-length ranking component in memory: the same-start, later-start and greedy-trade-off cases failed as expected, while the old earlier-start fixture still passed. No production ranking change was needed. At that commit: 63 focused candidate/admission/lifecycle tests and the full 1,833-test suite passed, along with |
a310f56 to
ed60280
Compare
| before_matches = [start for start in starts if view_text[:start].endswith(context_before)] | ||
| after_matches = [start for start in starts if view_text[start + len(value) :].startswith(context_after)] |
There was a problem hiding this comment.
_ground_item copies the source prefix and suffix for every occurrence of a suggested value, then repeats those copies when building matches. On long records containing the same name many times, this adds work proportional to the record length times the occurrence count and makes users wait longer for augmentation.
Use the position arguments of str.endswith and str.startswith in both passes instead of slicing the source.
There was a problem hiding this comment.
Fixed in 399ede8. Both context-presence checks and final match selection now use bounded str.endswith/str.startswith calls, avoiding source-prefix/suffix copies for each occurrence. Matching rules, failure categories, and admission policy are unchanged.
Added regressions for document-start/end contexts, overlong contexts, Unicode, and all 2,000 occurrences in a repetitive record. An offline comparison against the previous implementation produced identical outcomes across 65,520 synthetic cases. A local 104,000-character/8,000-occurrence microbenchmark (median of five runs) improved from approximately 23 ms to 2 ms; this is a grounding-only synthetic measurement, not an end-to-end throughput claim.
68 focused augmentation tests, all 1,844 suite tests, and make check passed.
|
Following up on the documentation review: #305 (comment) Updated in fa0a84d, and replaced the stale plan-only PR description with the implemented behavior. The guide now covers occurrence-specific detection (no generic propagation or name splitting), exact grounding versus semantic boundaries, greedy overlap selection, fallback versus record failure, duplicate-alias handling, tolerant augmenter extras, public source/score projection, internal trace diagnostics, and exported-builder/async-executor compatibility. The plan labels the old workflow as a historical baseline and records the retained seed-aware/tagged-prompt limitation. #267 remains the separately tested plain-text follow-up targeting this branch; #369 and #372 are not claimed implemented here. The inspection tutorial source and notebook Markdown are synchronized. No notebook model calls were executed: code cells, metadata, and cached outputs are unchanged, and those outputs are not new evidence for this revision. Strict docs build and make check passed. |
|
@binaryaaron — responses to the remaining review comments after the fixes, cleanup, and restack: C2: tagged augmenter context#267 is now restacked at C5: #349 compatibilityBoth identified causes are addressed: C7: excluded-label synonymsTracked separately in #369. Exact normalized-label eligibility remains enforced here; semantic synonyms in permissive augmentation are not solved by that check. Prompt guidance about excluded types/close synonyms remains follow-up work, not a deterministic synonym guarantee or an implemented fix in #305. C9: test/plan volume
C10: representation/admission policyThe main rebase removes the legacy synchronous validator dispatcher/factory and bridge; validation and augmentation plugins now execute asynchronously. The broader EntitySpan-plus-flag rewrite is not included. In particular, treating a duplicate assigned alias as an omitted decision would change the reviewed admission policy: we intentionally retain response rejection, bounded semantic retry/failover, and terminal record failure on exhaustion. Unknown aliases remain ignored, and missing/unusable unique decisions in otherwise valid responses retain eligible exact-span fallback. We want to preserve explicit decision/admission models for future validation work, rather than change that policy as cleanup. Representation simplification can be evaluated separately without weakening those distinctions. Current #305 validation: 1,844 tests, make check, and strict docs build passed locally. CI/reviewer approval remain separate gates. |
C1 is the remaining concern for me. I’m comfortable with (b), provided #267 lands with this and we compare the combined behavior against main before merging. The selected-miss replay doesn’t tell us how much recall we lose by removing propagation. In the scripted Redact check, the second That comparison should cover repeated sensitive mentions, standalone name parts, and identical text with different meanings, using independent annotations. Keep the models, providers, and settings fixed, and report recall, exposed mentions, false positives, and record failures. Include the Gemma organization control once a sanitized fixture is available. The other blocking comments look addressed at C7/C9/C10 can follow separately. Any simplification should preserve rejection and retries for duplicate aliases. One small description fix: #267 is already restacked, so that sentence can be updated. |
At Dropping only that column lets both proceed. The same input succeeds in preview. Please make the inventory safe across the batch write/read/write boundary while retaining its diagnostic value. The regression should carry real batch-loaded detection output into Substitute and latent detection. One smaller documentation correction: the failed-record examples in |
399ede8 to
28f8439
Compare
|
Detection-stack evaluation: #305 + #267 + #335The detection pipeline has three steps: GLiNER finds candidate names, IDs and other sensitive text → an LLM accepts or rejects those candidates → an LLM augmenter looks for what was missed. The combined changes keep decisions tied to specific text occurrences rather than assuming identical strings always have the same meaning. Main is the baseline code; stack is the combined set of PRs named above. We evaluated seven datasets: OpenPII, PrivacyBench, Tau-bench retail conversations, legal documents, Nemotron PII, a transcript/repeated-entity regression set, and biographies. GLiNER processes overlapping windows of input words. 128/32 means a 128-word window with 32-word overlap; 384/128 means a 384-word window with 128-word overlap. These are settings for the GLiNER detector, not for the LLM validator or augmenter. The results show fewer clear contextual mistakes and comparable or improved OpenPII annotation agreement in the tested settings. They also identify three remaining issues: personal-name coverage in Legal, repeated order-ID coverage on Tau-bench, and increased processing time. These measurements evaluate the combined stack, not #305 in isolation; they do not establish that the stack preserves every sensitive occurrence. Runs use Annotate, which marks detected spans locally. The measured work covers detection, validation, augmentation and local annotation. Reading the tables: P is precision—how many predictions overlap the reference annotations; R is recall—how many reference spans are found. Both are percentages, pooled over matching returned records, without requiring identical labels or exact boundaries. They measure annotation agreement, not semantic perfection. Times are minutes:seconds for the whole requested batch, excluding queueing and model startup; they include pipeline overhead and internal retries. Cross-model serving/reasoning profiles differ, so compare main and stack within each row. Configurations: GLiNER confidence threshold is 0.3 throughout. Main and stack match within each comparison. The 128/32 Glimmer/Nemotron evaluation covers OpenPII, PrivacyBench, Tau-bench and legal—not the other three datasets. PrivacyBench and Tau-bench use tailored labels and summaries; OpenPII/legal retain default label policies. Each dataset section labels its configuration explicitly. 1. OpenPII — default label policy, 128/32
Result. Glimmer improves both precision (96.32 → 96.81) and recall (90.75 → 91.85). Nemotron has a small tradeoff: precision 95.60 → 95.39, recall 92.16 → 92.32, F1 93.85 → 93.83. This supports comparable or improved annotation agreement in these settings, without establishing statistical equivalence. Do the differences matter? The reference includes disputed measurement labels, unfilled placeholders and honorifics not explicitly requested by the detection policy. Such mismatches do not all represent privacy leaks, but genuine missed names/identifiers still matter. Reading the original OpenPII text alongside the predicted spans and supplied annotations also revealed differences in entity labels and where a span starts or ends. Overlap scoring does not distinguish all of these differences. The scores retain the supplied annotations. Manual observations explain individual disagreements without treating genuine identity misses as harmless. Supplementary models — broad default policy, 384/128
GPT-OSS improves both P/R; Gemma and Qwen trade some recall for precision. Manual review of the broad-profile runs found concrete cleanup examples: a section-heading number is no longer treated as age, and a surname interpretation is not copied onto an ordinary word with the same spelling. Those observations support the contextual cleanup in that configuration; they are not a fresh manual audit of every 128/32 prediction. 2. PrivacyBench — six identity labels and a task-specific summary, 128/32
Result. Glimmer trades about one precision point for one recall point, with F1 97.94 → 97.96. Nemotron has unchanged recall and lower precision, with F1 97.94 → 97.44. These are useful identity-focused results, with a real precision cost rather than uniform improvement. Manual interpretation of the Glimmer 128/32 pair. Some predictions unmatched by gold are valid contextual nicknames/initials omitted from the reference. One additional stack prediction is an actual policy error: an ordinary conferencing product/service is labeled as a company. Genuine nickname/identity misses also remain. Thus neither “all extra detections are false positives” nor “the precision loss is entirely annotation noise” is supported. The six labels cover names, email, handles and named companies/organizations. The summary preserves incidental times, amounts, generic departments and ordinary product references unless they form part of a selected identity. Supplementary models — broad default policy, 384/128
This broad policy includes categories outside the narrow identity reference, so its scores cannot be pooled with the six-label comparison. Across the broad-profile review, 113 of 410 distinct detections were literal categories outside the reference scope and 13 were clear wrong-meaning detections. Configuration mismatch explains substantial measured over-detection, but not all mistakes or genuine identity losses. 3. Tau-bench retail conversations — 16 retail labels and a task-specific summary, 128/32
Result. Payment-ID occurrence coverage is complete on these matching traces; order-ID coverage remains lower in stack. These are mechanical checks of explicit field values at their source occurrences, not precision/recall over all personal information. Missing output rows are excluded from the matched comparison and remain visible in delivery counts. Why it matters. Actual customer-linked order/payment IDs must be protected wherever repeated. Product/item catalog IDs should be preserved. All 41 distinct observed order IDs use Supporting manual observations — broad default policy, 384/128On matching traces, manually reviewed clear semantic-error counts decrease 84 → 32 for Gemma and 69 → 29 for GPT-OSS. Glimmer HIGH has zero errors in this narrow category in both arms; that is not a claim of perfect labels, boundaries or recall. The same broad configuration has worse repeated-ID coverage. This supports contextual cleanup while preserving the separate occurrence-recovery concern. Those review counts are not measurements of the tailored 128/32 profile. 4. Legal — default label policy, 128/32We read all 25 legal records and compared both models’ main/stack outputs. The counts below use the same returned documents within each model: 25 for Glimmer and 24 for Nemotron. Nemotron’s missing stack record is reported separately from entity misses.
What improves. Both stack models make fewer reviewed wrong-meaning detections. Examples include no longer mistaking a section marker for a person's name, or a treaty for an organization in some documents. All returned outputs protect the primary application number. Generic court/government references, dates and actual job titles are treated separately as policy-dependent information, not automatically counted as mistakes. What still matters. The known six-later-surname example is covered 6/6 by all four outputs, but another document exposes a remaining repeat-coverage loss: both main models protect a lawyer's surname at all three occurrences; both stack models protect the first occurrence and miss the two later ones. Glimmer stack also loses several lawyer initials and one given-name component, while recovering initials in other examples. Nemotron's boundary/type-error count increases despite fewer wrong-meaning detections. The Legal results show that name handling needs to improve: correct treatment of one occurrence does not reliably protect later mentions, and initials and given-name components can still be lost. This is a personal-name coverage problem, separate from the Tau-bench order/payment-ID patterns targeted by regex. The table shows selected manual error categories, not a false-positive rate. There is no exhaustive legal gold reference, so these counts and specific coverage checks are not whole-dataset precision/recall. They describe final-pipeline outputs and do not assign credit or blame to GLiNER, validation or augmentation individually. Legal review, example by example — Glimmer and Nemotron, 128/32Documents are numbered 01–25 in input order. Error counts below cover reviewed wrong-meaning predictions, not all policy-dependent detections or a false-positive rate. Initials are genuine identity text, even when the full surname elsewhere is already protected. “Not returned” is a missing record, not zero errors.
The review checks every full document and all returned predictions, reusing exact prior span judgments only after source/hash verification. One source text appears under two record IDs; both evaluated records are retained. Absence of a reviewed name miss is not proof of exhaustive recall. 5. Nemotron PII — broad default policy, 384/128
Result. Qwen and Nemotron improve both P/R; Glimmer trades precision for recall; Gemma trades a little recall for precision; GPT-OSS is nearly flat. This dataset was not part of the 128/32 evaluation. Annotation interpretation. Manual review found genuine overreach on empty headings, unselected form options and generic roles. It also found source-supported medical information and credentials absent from the reference. Consequently, lower precision can reflect either unnecessary detection or protection beyond incomplete gold. Actual populated identities/credentials must remain protected; label/boundary disagreements should be distinguished from privacy misses. The review does not establish that every small recall loss is harmless. 6. Transcript/repeated-entity regression — broad default policy, 384/128
Result. All arms return 17 transcripts, and every model has fewer manually reviewed wrong-meaning detections—for example, speaker markers interpreted as names and generic advice interpreted as personal attributes. This is direct evidence of contextual cleanup. Remaining cost. The 19 targets are selective positives, not exhaustive gold. Some lost occurrences are genuine repeated names: in a reviewed Qwen example, main protects eight surname occurrences and stack protects two. Fewer false alarms can coexist with worse name coverage. This dataset was not evaluated with 128/32 windows. 7. Biographies — broad default policy, 384/128
Result. The stack reduces some over-interpretation, including work topics mislabeled as education for Glimmer, GPT-OSS and Nemotron. These counts describe one manually reviewed error category, not precision or unchanged recall. There is no exhaustive gold reference or 128/32 evaluation here. Genuine employers, birthplace and education can identify a person; a background institution or discussion topic need not be a personal attribute. That distinction requires an explicit protection policy rather than suppressing all locations or organizations. Follow-ups
Augmenter follow-up evidence — separate prompt comparison, 128/32An accepted-detections-aware prompt improves PrivacyBench annotation agreement for both models to P/R 98.97/97.97. OpenPII is mixed: Glimmer changes 96.81/91.85 → 95.71/92.32, while Nemotron changes 95.39/92.32 → 96.61/92.79. These compare the existing stack prompt with the revised prompt; they are not the main/stack results above. The revised prompt is not ready as a blanket replacement: Nemotron Tau returns 16/25, with nine augmentation token-limit cutoffs. On the same 16 returned traces, order coverage declines 217/222 → 206/222. Strengthening augmentation must improve useful occurrence coverage without losing records or expanding generation cost unchecked. The complete gold-mismatch review for revised-prompt OpenPII does not amount to a manual audit of all existing-prompt 128/32 predictions. Methodology and interpretation limitsEvaluated source snapshots: main Frozen inputs, order, model settings and sampling are held fixed within each main/stack pair (threshold 0.3, seed 42). Changing windows, labels or input summaries creates a separate configuration comparison; differences across those groups cannot be attributed solely to model choice or a PR. The broad-profile manual review covered 392 source records, 13,669 distinct labeled predictions and 1,775 reference contexts, including supplementary model configurations. Those contextual judgments are not independent exhaustive gold. The Legal 128/32 review covers all 25 full documents and the 99 returned main/stack outputs. Of 1,554 distinct labeled predictions, 1,355 reused exact prior judgments after frozen-source verification and 199 received new contextual judgments. The targeted PrivacyBench review and mechanical Tau-bench checks have their own stated scopes. Reported P/R uses the original annotations; manual judgments are reported separately. Selected transcript targets and explicit retail-field checks cannot support exhaustive precision/F1. “—” is unavailable, not zero detections. Batch timing covers all requested inputs, while gold scores use matching returned records. The annotated benchmarks have 100 requested records; document/retail sets have 25; the transcript regression set has 17. Runtime tables describe single observed runs, not replicated performance estimates. Model architectures, reasoning modes, GPU layouts and concurrency differ across models; main/stack comparisons within a row are the relevant controlled comparison. |
28f8439 to
4c88896
Compare
Signed-off-by: Lipika Ramaswamy <lipika.ramaswamy@gmail.com>
Preserve occurrence identities and request-scoped validation admission, ground augmentation against source context, and reconcile candidates deterministically. Retain GLiNER alternatives, async plugin tracing, and nested lifecycle storage contracts. Include regression coverage and user-facing detection documentation. Signed-off-by: Lipika Ramaswamy <lipika.ramaswamy@gmail.com>
4c88896 to
6352cba
Compare
|
The CI test failures are due to upstream NVIDIA-NeMo/DataDesigner#1143. Its Arrow-backed reader can corrupt valid nested arrays during batch reads, causing With the exact fix from NVIDIA-NeMo/DataDesigner#1142 applied as a local source overlay, the full suite passes: 2,017 passed, 2 skipped. This PR is gated on consuming a released or pinned DataDesigner dependency containing that fix. No downstream workaround is included. |
Build regex detection on PR #305. Preserve bounded matching, local validators, per-rule admission routes and regex-only labels while using the shared observation admission and reconciliation path. Keep rule claims independent until admission and require exact accepted regex evidence for model-derived regex-only labels. Do not restore value propagation or replay validator decisions during finalization. Signed-off-by: lipikaramaswamy <lramaswamy@nvidia.com>
|
Related Issue
Related to #262.
Merge gate
Blocked on NVIDIA-NeMo/DataDesigner#1142 (fixes NVIDIA-NeMo/DataDesigner#1143), and consuming a released or pinned dependency containing that fix. DataDesigner 0.10.0's default Arrow-backed reader can corrupt valid nested lifecycle columns during batch reads. This PR does not include a downstream workaround.
Summary
Implements a shared, deterministic detection-candidate lifecycle before the regex integration in #265. The design document retains the rationale and historical baseline; the detection guide documents the implemented behavior.
accepted,rejected, orfallback_protectedadmission states and are not replayed over independently admitted augmentation output.Failure and Admission Nuance
Compatibility and Follow-ups
Detectconfiguration knobs, but detection behavior changes: no automatic propagation/name parts; eligible labels use normalized matching; fallback-only public entities havesource="fallback"; absent native scores publish0.0rather than synthetic augmenter confidence1.0.final_entitiesshape remains the entity output. Internal trace structures now expose observations, request inventories, admission states, candidates, and augmentation attempt diagnostics. The public representativesourceis not the complete accepted-provenance ledger.Type of Change
Validation
main. Full suite with the exact DataDesigner#1142 source overlay (1c606c44d1326e1eb8b8bb6bd36472b260b814e1): 2,017 passed, 2 skipped. This validates integration with the upstream fix; stock DataDesigner 0.10.0 still fails the seven affected full-run integration tests, so the merge gate above remains.make check: format/lint, typing, lockfile, and SPDX checks passed.make docs-build: strict documentation build passed.git diff --check: passed.make convert-notebooks) was intentionally not run because it executes model calls. Cached outputs are not new results from this revision.Contributor Checklist
PR stack
Merge/review order: #305 → #267 → #335.
main.After a parent is squash-merged, restack the remaining child commits onto the updated parent/
mainand retarget their PRs as appropriate. These links describe dependency order, not approval or CI status.