Repository navigation
Conversation
A model entry in the spaCy NLP engine configuration can now set `max_length`. SpacyNlpEngine sets it as `nlp.max_length` after it loads the model, so spaCy rejects a longer text (E088) before the pipeline runs. Without the key, spaCy's default is kept. `/analyze` now returns 413 with a clear message when spaCy rejects a text as too long, in place of the generic 500. Refs data-privacy-stack#2446
Build the message without data from the exception, so that no part of the exception text reaches the client. The exception is used only to detect spaCy's E088.
…NLP model - New page "Analyzing long texts": split the text before calling `analyze`, with an example and its limits, and pointers to `max_length` and to the no-op NLP engine. - New section "Running without an NLP model" on `NoOpNlpEngine`: how to set it up, which entities are lost, that context words in the text no longer raise scores, and what this means for score thresholds. Refs data-privacy-stack#2446
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
This is item 3 of the direction agreed in #2446, the docs that follow #2517. It changes documentation only.
Draft until #2517 merges. The branch is built on top of #2517, because the new page links to the
max_lengthsection that #2517 adds. Until then the diff also shows the two commits of #2517. Only the last commit,docs(analyzer): add guidance on long texts and on running without an NLP model, belongs to this PR (3 files underdocs/andmkdocs.yml). After #2517 merges I will rebase this branch ontomainand mark it ready.New page "Analyzing long texts" (
docs/analyzer/long_texts.md, added to the Presidio Analyzer navigation):analyze.CharacterBasedTextChunker.predict_with_chunking, which shifts the results to the positions of the full text and removes duplicates from the overlap.max_lengthand the413response from feat(analyzer): expose spaCy max_length in the NLP engine config #2517, to theNoOpNlpEnginesection, and totext_chunkerfor the recognizers that run their own model.New section "Running without an NLP model" in
docs/analyzer/customizing_nlp_models.md:NoOpNlpEnginefits, and how to set it up in YAML, on the server (NLP_CONF_FILE), and in code.PERSON,LOCATION,NRPwith the default recognizers), and the score boost from context words in the text.NoOpNlpEngine.contextparameter.How I checked the content:
main.analyzecall on a 23,000-character text with 958 results. With an overlap of 100 to 200 characters, 957 or all 958 results were the same. The one that differed was a date at a chunk edge that was detected only in part. Without overlap, 4 to 5 results at chunk edges were split in two. The page states these limits.NLP_CONF_FILE=presidio_analyzer/conf/no_op.yaml.Behavior change: none, documentation only.
One thing to confirm: the page recommends
CharacterBasedTextChunkerfor splitting at the analyzer level. It was written for the NER recognizers, but it is exported frompresidio_analyzer.chunkersand it works for this use. If you prefer a plain loop over chunks in the example, I can change it.Issue reference
Part of #2446 (item 3). Depends on #2517.
Checklist
RecognizerRegistryProviderNo code is changed, so there are no new tests.