Repository navigation
fix(analyzer): keep Stanza multi-word tokens as surface tokens to preserve text and entity offsets - #2253
fix(analyzer): keep Stanza multi-word tokens as surface tokens to preserve text and entity offsets#2253ManoharPaturi wants to merge 15 commits into
Conversation
…serve text and entity offsets Stanza's mwt processor expands German contractions (im/am/zum/zur/beim/ vom/ins/ans) into multiple words. StanzaTokenizer.__get_tokens_with_heads flattened the expanded words, which broke alignment with the original text: _convert_doc then replaced the doc text with space-separated expanded tokens, silently dropped every NER entity (offsets no longer mapped to tokens), and, on indented text, let tokens_indices end before pattern matches near the end of the text, raising 'Did not find word ...' in LemmaContextAwareEnhancer (HTTP 500). Keep a multi-word token as a single surface token (MultiWordTokenSurface: text/lemma are the surface form, other annotations come from the first expanded word) and remap 1-based word heads to the collapsed token list so dependency heads around the collapsed token stay correct. Fixes data-privacy-stack#2249
|
thanks a lot for pulling the branch and testing it yourself, glad it holds up across the contractions. appreciate the detailed writeup. |
|
@omri374 removed, thanks. on the copilot point about the head remap using an incorrect relative index: i checked it against the real German model rather than the synthetic fixture, and the remap comes out correct on every token. for 'Wir treffen uns im Büro mit Thomas Bergmann.' stanza gives words (in, dem) as ids 4,5 both governed by id 6 (Büro), which collapses to token 3 governed by token 4:
the offset + word_index_to_token_index term is the governor's absolute doc index and len(tokens) is the current token's absolute index at that point, so the difference is the relative head. happy to add this sentence as a regression test if useful. |
omri374
left a comment
There was a problem hiding this comment.
Thanks! The only thing missing is proper handling of downloading Stanza models during unit tests. You'd might need to update the conftest here:
https://github.com/ManoharPaturi/presidio/blob/b71a4b4aeaa889a0f51eb2c7d0612fc1702b1619/presidio-analyzer/tests/conftest.py#L31
To add stanza_de as a preloaded NLP Engine
Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com> Co-authored-by: Arjun Sanjay Pakhan <arjunpakhan@gmail.com>
|
done, trailer added to the final commit. thanks for the thorough verification pass, that kind of pull-the-branch testing is exactly what the fix needed. |
3c79b36 to
6bdb5c6
Compare
…test Remove the German Stanza model download and the opt-in env flag from conftest.py. Replace the live German integration tests in test_stanza_nlp_engine.py with a standalone unit test using a real Stanza Document generated via CoNLL.conll2doc, covering multi-word token collapse, surface text offsets, dependency heads, and entity character offsets without model download overhead. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
|
@omri374 Updated per your recommendation. The German model download, engine registration, and opt-in env flag have been completely removed from conftest.py, and the live German tests were replaced with your CoNLL-U synthetic document unit test in test_stanza_nlp_engine.py. It exercises _convert_doc with real Stanza Token, Word, and Span objects across the multi-word token collapse, surface text offsets, dependency heads, and NER entity boundaries without any model download overhead. Ruff checks pass cleanly. |
The public StanzaNlpEngine.load() and process_text() path now has a regression with the pipeline mocked at load_pipeline: no model download, the real Stanza Document from CoNLL-U converted by the real StanzaTokenizer, asserting the preserved surface text, token offsets and mapped entity offsets. Fails on the unpatched conversion. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
Head branch was pushed to by a user without write access
|
The coverage gap is filled in 2dbfb8d: the German multi-word token now also has a regression through the public engine path. StanzaNlpEngine.load() and process_text() run end to end with the pipeline mocked at load_pipeline, so no model is downloaded (keeping the no test overhead constraint from review): the mocked language object converts the real Stanza Document built from CoNLL-U through the real StanzaTokenizer, which is what the spacy-stanza pipeline would return for this text. The test asserts the surface text is preserved, every token index slices its text out of the original string, and the PER entity comes back mapped to PERSON with its original character offsets 28 to 43. On the unpatched conversion the test fails with the corrupted document text from the issue repro, so it is a real regression for the reported bug, not just for the helper. |
There was a problem hiding this comment.
🔵 Needs a closer look
A critical dependency-head indexing bug remains: the remapped governor is absolute while conversion expects a relative offset, causing the new test to fail and producing incorrect heads for non-root tokens.
No open issues 🎉
1 resolved since last review
Review effort: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
A critical active finding remains: the remapped dependency head uses an absolute index instead of the required relative delta, causing the regression test to fail and producing incorrect dependency heads.
No open issues 🎉
Review effort: Lite

Fixes #2249.
StanzaTokenizer.__get_tokens_with_headsflattened stanza multi-word-token expansions (im->in+dem), so token texts no longer matched the source text._convert_docthen rebuilt the doc text from those tokens, which corrupted the document text (see the issue repro), made Stanza NER spans miss every token boundary so entities were silently dropped, and could raiseValueErrorfrom the lemma enhancer on indented text.Multi-word tokens now stay a single surface token (a small wrapper carrying surface text/lemma plus the first expanded word's annotations), with a word-index to token-index map that remaps dependency heads across the collapsed token. Single-word tokens keep the previous code path byte-for-byte.
Verified with the real German model:
doc.textis preserved,PERandIP_ADDRESSentities are detected where pristine drops them. 11 new tests (pure-unit head remap + German model tests over all 8 contractions) fail on main and pass here; existing stanza suites pass unchanged (12 + 13 + 16), ruff clean vs main baseline.