Skip to content

fix(analyzer): keep Stanza multi-word tokens as surface tokens to preserve text and entity offsets - #2253

Open
ManoharPaturi wants to merge 15 commits into
data-privacy-stack:mainfrom
ManoharPaturi:fix/stanza-mwt-offsets
Open

ManoharPaturi wants to merge 15 commits into
data-privacy-stack:mainfrom
ManoharPaturi:fix/stanza-mwt-offsets

Conversation

@ManoharPaturi

Copy link
Copy Markdown

Fixes #2249.

StanzaTokenizer.__get_tokens_with_heads flattened stanza multi-word-token expansions (im -> in + dem), so token texts no longer matched the source text. _convert_doc then rebuilt the doc text from those tokens, which corrupted the document text (see the issue repro), made Stanza NER spans miss every token boundary so entities were silently dropped, and could raise ValueError from the lemma enhancer on indented text.

Multi-word tokens now stay a single surface token (a small wrapper carrying surface text/lemma plus the first expanded word's annotations), with a word-index to token-index map that remaps dependency heads across the collapsed token. Single-word tokens keep the previous code path byte-for-byte.

Verified with the real German model: doc.text is preserved, PER and IP_ADDRESS entities are detected where pristine drops them. 11 new tests (pure-unit head remap + German model tests over all 8 contractions) fail on main and pass here; existing stanza suites pass unchanged (12 + 13 + 16), ruff clean vs main baseline.

…serve text and entity offsets

Stanza's mwt processor expands German contractions (im/am/zum/zur/beim/
vom/ins/ans) into multiple words. StanzaTokenizer.__get_tokens_with_heads
flattened the expanded words, which broke alignment with the original
text: _convert_doc then replaced the doc text with space-separated
expanded tokens, silently dropped every NER entity (offsets no longer
mapped to tokens), and, on indented text, let tokens_indices end before
pattern matches near the end of the text, raising 'Did not find word ...'
in LemmaContextAwareEnhancer (HTTP 500).

Keep a multi-word token as a single surface token (MultiWordTokenSurface:
text/lemma are the surface form, other annotations come from the first
expanded word) and remap 1-based word heads to the collapsed token list
so dependency heads around the collapsed token stay correct.

Fixes data-privacy-stack#2249
Copilot AI lite review requested due to automatic review settings September 13, 2026 16:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@ManoharPaturi

Copy link
Copy Markdown
Author

thanks a lot for pulling the branch and testing it yourself, glad it holds up across the contractions. appreciate the detailed writeup.

@omri374
omri374 requested a lite review from Copilot September 15, 2026 12:51
Comment thread FIX-SUMMARY.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Dependency-head remapping still uses an incorrect relative index.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

@ManoharPaturi

Copy link
Copy Markdown
Author

@omri374 removed, thanks.

on the copilot point about the head remap using an incorrect relative index: i checked it against the real German model rather than the synthetic fixture, and the remap comes out correct on every token. for 'Wir treffen uns im Büro mit Thomas Bergmann.' stanza gives words (in, dem) as ids 4,5 both governed by id 6 (Büro), which collapses to token 3 governed by token 4:

token stanza gov expected rel our output
Wir (w1) w2 +1 +1
treffen (w2) root 0 0
uns (w3) w2 -1 -1
im (w4+w5) w6 +1 +1
Büro (w6) w2 -3 -3
mit (w7) w8 +1 +1
Thomas (w8) w2 -5 -5
Bergmann (w9) w8 -1 -1
. (w10) w2 -7 -7

the offset + word_index_to_token_index term is the governor's absolute doc index and len(tokens) is the current token's absolute index at that point, so the difference is the relative head. happy to add this sentence as a regression test if useful.

Copilot AI review requested due to automatic review settings September 15, 2026 15:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@omri374 omri374 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@omri374 omri374 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! The only thing missing is proper handling of downloading Stanza models during unit tests. You'd might need to update the conftest here:
https://github.com/ManoharPaturi/presidio/blob/b71a4b4aeaa889a0f51eb2c7d0612fc1702b1619/presidio-analyzer/tests/conftest.py#L31

To add stanza_de as a preloaded NLP Engine

Copilot AI review requested due to automatic review settings September 16, 2026 10:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
Co-authored-by: Arjun Sanjay Pakhan <arjunpakhan@gmail.com>
@ManoharPaturi

Copy link
Copy Markdown
Author

done, trailer added to the final commit. thanks for the thorough verification pass, that kind of pull-the-branch testing is exactly what the fix needed.

Copilot AI review requested due to automatic review settings September 16, 2026 15:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings September 23, 2026 12:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread presidio-analyzer/tests/conftest.py Resolved
…test

Remove the German Stanza model download and the opt-in env flag from
conftest.py. Replace the live German integration tests in
test_stanza_nlp_engine.py with a standalone unit test using a real Stanza
Document generated via CoNLL.conll2doc, covering multi-word token collapse,
surface text offsets, dependency heads, and entity character offsets without
model download overhead.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 23, 2026 14:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@ManoharPaturi

Copy link
Copy Markdown
Author

@omri374 Updated per your recommendation. The German model download, engine registration, and opt-in env flag have been completely removed from conftest.py, and the live German tests were replaced with your CoNLL-U synthetic document unit test in test_stanza_nlp_engine.py. It exercises _convert_doc with real Stanza Token, Word, and Span objects across the multi-word token collapse, surface text offsets, dependency heads, and NER entity boundaries without any model download overhead. Ruff checks pass cleanly.

Copilot AI review requested due to automatic review settings September 24, 2026 18:01
omri374
omri374 previously approved these changes Sep 24, 2026

@omri374 omri374 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@omri374
omri374 enabled auto-merge (squash) October 24, 2026 12:28
Copilot AI lite review requested due to automatic review settings November 19, 2026 08:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical indexing bug corrupts dependency-head offsets for non-root tokens, and the regression test does not cover the real German MWT conversion path or entity-offset preservation.

1 open issue

Review effort: Lite

The public StanzaNlpEngine.load() and process_text() path now has a
regression with the pipeline mocked at load_pipeline: no model download,
the real Stanza Document from CoNLL-U converted by the real
StanzaTokenizer, asserting the preserved surface text, token offsets and
mapped entity offsets. Fails on the unpatched conversion.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
auto-merge was automatically disabled November 19, 2026 11:10

Head branch was pushed to by a user without write access

Copilot AI lite review requested due to automatic review settings November 19, 2026 11:10
@ManoharPaturi

Copy link
Copy Markdown
Author

The coverage gap is filled in 2dbfb8d: the German multi-word token now also has a regression through the public engine path. StanzaNlpEngine.load() and process_text() run end to end with the pipeline mocked at load_pipeline, so no model is downloaded (keeping the no test overhead constraint from review): the mocked language object converts the real Stanza Document built from CoNLL-U through the real StanzaTokenizer, which is what the spacy-stanza pipeline would return for this text.

The test asserts the surface text is preserved, every token index slices its text out of the original string, and the PER entity comes back mapped to PERSON with its original character offsets 28 to 43. On the unpatched conversion the test fails with the corrupted document text from the issue repro, so it is a real regression for the reported bug, not just for the helper.

@ManoharPaturi

Copy link
Copy Markdown
Author

@omri374 the review comment is resolved (public path regression added in 2dbfb8d) and the branch is merged with current main in db96d0c. The auto merge seems to have been cancelled along the way, so the workflows will need your approval again whenever convenient.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A critical dependency-head indexing bug remains: the remapped governor is absolute while conversion expects a relative offset, causing the new test to fail and producing incorrect heads for non-root tokens.

No open issues 🎉

1 resolved since last review

Review effort: Lite

Copilot AI lite review requested due to automatic review settings November 19, 2026 11:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A critical active finding remains: the remapped dependency head uses an absolute index instead of the required relative delta, causing the regression test to fail and producing incorrect dependency heads.

No open issues 🎉

Review effort: Lite

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StanzaNlpEngine: German multi-word tokens (im/am/zum) replace the doc text and drop every NER entity; indented text ends in 500 "Did not find word"

3 participants