[asteier2026]

Summary

Stacked on #305 (lipikaramaswamy/feature/detection-semantics, current parent 399ede83). Depends on #305; retarget to main after that PR lands.

The augmenter now receives canonical plain source text (COL_TEXT), without engine tags, seed-entity lists, or instructions to avoid existing detections. Both label-mode examples use plain text and retain value, label, context_before, context_after, and reason.

Preserves #305's exact-source grounding, bounded grounding retries, label scope, date policy, independent seed admission, and shared candidate reconciliation. Removes augmentation's obsolete tagged-input dependencies and tag-stripping guidance, including the correction prompt. The old #267 resolve_overlaps() patch is intentionally not ported: duplicate and conflicting discoveries follow #305's existing policy. No every-occurrence instruction or native schema enforcement is added. Public API is unchanged.

Plan and experiment details: plans/267/plain-text-augmentation.md. Tracks #262; separate from the regex work in #265.

Reported replay evidence

These are supplied experiment results from a prompt-only adaptation onto 97c0ae80, not new measurements on this head. Augmentation-only replays retained original validated seeds, sampling, and the 32,768-token output limit; no fresh detector or validator calls were made.

Model	Selected known-miss checks: current → plain text	Previously correct checks
Luna	6/9 → 9/9	15/15 → 15/15
Nemotron 3.5 Super VL	7/16 → 15/16	9/9 → 9/9
Gemma 4 26B A4B	16/45 → 45/45	6/6 → 3/6

Checks include repeats, not unique-record counts. All 34 Luna/Super attempts returned. Gemma returned 48/51 after failed-attempt recovery on another provider; three still failed output-format validation. Returned replays had no invalid final source spans. The Gemma regression was an organization mention omitted in all three repeats, not a finding discarded by reconciliation.

Average augmentation time including retries rose approximately 19 → 28 seconds for Luna and 19 → 32 seconds for Super. Gemma timing is confounded by concurrent endpoint traffic and provider changes.

This small selected failure-enriched cohort reused saved baselines and is not a fresh counterbalanced comparison. It does not establish population precision/recall, correctness of every additional prediction, or a general quality/performance improvement. A fresh paired provider-backed comparison and investigation of the exact Gemma control remain outstanding; no broad improvement claim is made.

Validation

• Offline strict/permissive prompt rendering and exact-source example grounding.
• Sync/async plain-source rendering despite overlapping seeds, plus preservation of literal source markup.
• Duplicate rediscovery/provenance and conflicting label/bounds behavior through shared reconciliation.
• Synthetic organization control distinguishing model omission from reconciliation loss; exact reported fixture unavailable locally.
• 174 focused tests passed; full suite: 1,825 passed with TMPDIR=/tmp and a short pytest base directory to avoid unrelated macOS temporary-path failures.
• make check passed in an isolated dev/docs environment (the local notebook extras expose existing unused-ignore warnings in refactor(detection): establish deterministic entity candidate lifecycle #305). Strict make docs-build passed.
• No real PII fixtures, credentials, public API changes, or bundled skill changes.

PR stack

Merge/review order: #305 → #267 → #335.

• refactor(detection): establish deterministic entity candidate lifecycle #305: candidate-lifecycle foundation, targeting main.
• fix(detection): augment plain source on candidate lifecycle #267: plain-source augmentation, stacked on refactor(detection): establish deterministic entity candidate lifecycle #305.
• fix(detection): clarify partial calendar date validation #335: partial-calendar-date guidance, stacked on fix(detection): augment plain source on candidate lifecycle #267.
• feat(detection): add regex entity detection #265: separate regex feature built on refactor(detection): establish deterministic entity candidate lifecycle #305; it is not a prerequisite or merge gate for refactor(detection): establish deterministic entity candidate lifecycle #305, fix(detection): augment plain source on candidate lifecycle #267, or fix(detection): clarify partial calendar date validation #335.

After a parent is squash-merged, restack the remaining child commits onto the updated parent/main and retarget their PRs as appropriate. These links describe dependency order, not approval or CI status.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] This PR appears safe to merge; no new actionable issue was found.

Summary

Augmentation now reads canonical plain source text without detector tags or a seed-entity list.

• The augmenter finds entities in plain source text, whether or not they were detected already.

The earlier overlap concern is no longer outstanding. asteier2026 accepted the old behavior because augmentation was expected to return minimal spans; the current stack supersedes that behavior with shared longest-first selection.

_{Reviews (6) · Last reviewed commit: "fix(detection): augment plain source on ..." · Reviewed by Greptile}

[greptile-apps]

Broader sensitive spans discarded

When the augmenter identifies a broader sensitive value that overlaps a narrower detector span, this filter discards the entire augmenter entity instead of preserving its non-overlapping sensitive portion. For example, if the augmenter finds a full street address containing a detector-identified city, only the city reaches validation and replacement. The rest of the address can remain unanonymized.

How this was verified: Only entities retained by resolve_overlaps() are passed into validation and replacement, and no later stage reconstructs the discarded span or its residual intervals.

[asteier2026]

This is by design. If gliner finds Seattle and augment finds "near downtown Seattle", it's better to hold on to the validated gliner entity "Seattle" and throw out the rest. I think it's too messy to try and weed out parts of the augment span that should still be an entity. Augment is supposed to find minimal spans. If "Seattle, Washington" shows up, we want augment to tag "Seattle" and "Washington" individually.

[greptile-apps]

That makes sense. Given the augmenter’s minimal-span contract, dropping an overlapping augmenter result is intentional: the validated GLiNER span is preserved, while the augmenter is expected to emit separate minimal entities such as Seattle and Washington rather than a broader span containing Seattle. I’m withdrawing this concern.

[lipikaramaswamy]

Updating this older thread to reflect the current stack: #267 has been restacked at 16e9d0d onto #305 head 399ede8. The original seed-overlap rejection patch is no longer part of #267. It changes augmentation input/prompt only; independently grounded/admitted suggestions and detector observations flow through #305's shared eligibility, exact-identity coalescing, and final reconciliation.

Consequently, an eligible broader span is not discarded merely because it overlaps a narrower detector span: greedy longest-first selection prefers the broader supplied span in a containment conflict. Same-bound ties use acceptance/source policy. Partial overlaps still select one supplied span, not a synthetic union or residual intervals; this is not a maximum-total-coverage guarantee. Discriminating containment, later-start/longer, greedy-tradeoff, and excluded-fallback tests live in #305. The restacked #267 passed 1,841 tests, make check, and strict docs build. This supersedes the earlier seed-precedence explanation in this thread.

[lipikaramaswamy]

@asteier2026 — handoff for bringing this onto #305, including the local experiment results.

What we actually changed locally

We tested a prompt-only adaptation of this PR's a6f683a on the #305 implementation at 97c0ae80. Neither this PR's branch nor its base was changed; #267 still targets main.

The adaptation:

• Gives the augmenter canonical plain source text (COL_TEXT) instead of the engine-tagged text.
• Removes the existing/seed-entity list and the instruction not to repeat already-tagged entities.
• Removes overlap-group/tag-specific instructions and the guidance about stripping engine-inserted tags when copying text.
• Replaces the tagged example with a plain-text example, while preserving refactor(detection): establish deterministic entity candidate lifecycle #305's value, label, context_before, context_after and reason response fields.
• Keeps date guidance, label scope, exact-source grounding, grounding retries, admission and final candidate reconciliation unchanged.

We did not apply this PR's overlap-resolver change. #305 replaces that older path with occurrence-specific admitted candidates and shared final reconciliation. That part needs reassessment against #305, rather than carrying the old resolve_overlaps() modification forward unchanged. We also did not add an “every occurrence” instruction or native schema enforcement.

How we tested it

We replayed augmentation only, retaining each original run's exact validated seed candidates. No new detector or validator calls were made. Luna and Nemotron 3.5 Super VL used NVIDIA Inference; Gemma 4 26B A4B used OpenRouter. Sampling and the 32,768-token output limit were unchanged.

Model	Selected known-miss target checks protected: current → plain-text adaptation	Checks on previously correct examples
Luna	6/9 → 9/9	15/15 → 15/15
Super	7/16 → 15/16	9/9 → 9/9
Gemma	16/45 → 45/45	6/6 → 3/6

A target checked in three repeats contributes three checks; these are not counts of unique records. All 34 Luna/Super replay attempts returned. Gemma returned 48/51 after retrying only failed attempts on another provider; three responses still failed output-format validation. No invalid final source spans were found among returned replays.

The Gemma control regression was an organization mention omitted in all three repeats, not an accepted finding discarded during reconciliation. This needs investigation before claiming the prompt is an across-the-board improvement.

Average augmentation time, including retries, was approximately 19 → 28 seconds for Luna and 19 → 32 seconds for Super. We should not make a causal Gemma speed claim because its replay overlapped other endpoint traffic and failed attempts were recovered on another provider.

Limits: this was a small, selected failure-enriched cohort. We reused saved current-prompt outputs, rather than rerunning both prompts in a fresh counterbalanced comparison. The Gemma recovery also changed providers. These results show recovery of selected missed mentions, not population precision/recall or proof that all additional predictions are correct.

Suggested port

1. Rebase/retarget onto refactor(detection): establish deterministic entity candidate lifecycle #305 and apply the prompt/input changes to its current augmentation implementation, preserving the context-based response schema and date policy.
2. Retain refactor(detection): establish deterministic entity candidate lifecycle #305's source grounding, independent seed admission and final reconciliation; reconsider the old overlap patch separately.
3. Add offline tests for plain-text rendering (including strict/non-strict label modes), duplicate rediscovery, conflicting labels/spans, and source-exact occurrence grounding.
4. Include the Gemma control omission in a focused follow-up test and run a fresh paired prompt comparison before broad quality/performance claims.

The same results are recorded in the implementation-review comment on #305. This prompt change remains separate from #305 and from the regex work in #265.

[lipikaramaswamy]

Restacked onto the current #305 head, 399ede8. New #267 head: 16e9d0d; base remains lipikaramaswamy/feature/detection-semantics.

Only the original #267 commit was replayed. Range-diff shows the same intended patch (only a surrounding test-helper context changed), so this retains canonical plain-text augmentation, removal of seed/do-not-repeat instructions, and the existing grounding/admission/reconciliation contracts. It also inherits #305 fixes, async-only plugins, test cleanup, documentation, and the bounded grounding optimization. Original author/sign-off preserved; the old tip abb6d88 is retained locally for recovery.

Validation: all 1,841 tests passed; make check and strict make docs-build passed. No provider-backed inference or new quality benchmark was run; the earlier reported experiment remains historical evidence with its existing caveats.