Repository navigation
fix(analyzer): match each regex allow list term on its own - #2491
kushaldave002 wants to merge 2 commits into
Conversation
Joining the terms with | renumbers capture groups, so a backreference in a later term pointed at a group of an earlier term and the term stopped matching its own text. Compile and search each term separately.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
jamesconwaydev
left a comment
There was a problem hiding this comment.
Read it against main. The fix is right, and the test with the backreference in the second term shows exactly the failure from #2433.
One behaviour change I'd add to the description. The timeout used to bound one search per result, and now each term gets its own, so the worst case for a result is len(allow_list) * REGEX_TIMEOUT_SECONDS. With a few terms that's nothing. With a long list of patterns that each backtrack badly, it could multiply a slow analyze call that used to give up after one timeout. If that matters, one shared deadline per result would keep the old bound: compute the time left before each term and pass that as its timeout. Otherwise a sentence in the docs for allow_list_match="regex" would cover it.
Not a problem, just confirming: any() stops at the first term that matches, so a result that's allowed early costs one search, as before. Duplicate group names across terms used to share one numbering in the joined pattern, and now they can't collide at all. (Read only; not run.)
1AbdullahButt
left a comment
There was a problem hiding this comment.
Looks correct. Here's what I checked.
- This is the only place the allow list is joined.
"|".join(allow_list)appeared once onmain, atanalyzer_engine.py:572, and this PR replaces it (:590-592). Nothing else builds a pattern from the list. The image redactor's own OCR word check (image_analyzer_engine.py:174,:201) is a plainword not in allow_list, so no other call site needs the same fix. - The bug and the fix behave as described. With the default
DOTALL | MULTILINE | IGNORECASEflags and theregexmodule, the joined^(John) Smith$|^(\w\w)-\1$does not matchAA-AA, because\1points at(John). The second term compiled on its own does match it. The third test case works for the same reason: in the joined pattern,\2points at the(\d\d)group, which never takes part in the match. - It also fixes inline flags leaking between terms. In the
regexmodule's default version, a global inline flag such as(?-i)inside one term applied to the whole joined pattern. Compiling each term separately keeps it inside that term. - The keep/drop logic matches the old code.
any(...)over a generator (:598-601) stops at the first matching term, and a timed-out term counts as no match. A result is still kept if no term matches it, which is what the oldexcept TimeoutErrorbranch did. Empty terms are still filtered out first (:584-586). - Invalid patterns still fail early. All terms are compiled before the results loop, so a bad term raises
regex.errorin the same place as before, even if there are no results. git merge-treeagainstmainis clean.
The timeout is now per term instead of per result, so the worst case is len(allow_list) * REGEX_TIMEOUT_SECONDS per result. That was already raised in the earlier review, so I won't go over it again.
I didn't run the tests.
Reviewed with help from Claude Code.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Change Description
With
allow_list_match="regex",AnalyzerEngine._remove_allow_listjoined all terms into a single pattern with"|".join(allow_list). Capture groups are then numbered across the whole list, so a backreference in a later term points at a group of an earlier term. That term silently stops matching its own text and the result stays in the output with no error.This change compiles each term on its own and keeps a result out of the output if any term matches it. The per-term timeout guard is kept: a term that times out is logged and counts as not matching, so the result is kept unless another term matches.
Existing behavior that changes:
|, behave as before; with the joined pattern they could break other terms.Issue reference
Fixes #2433
Checklist
RecognizerRegistryProviderTesting
From
presidio-analyzer:uv run pytest tests/test_analyzer_engine.py tests/test_analyzer_request.py- 81 passeduv run ruff check .- passed;ruff format --checkon the two changed files - cleanmainfor two of its four cases (the reproduction from the issue and the term with its own top-level alternation) and passes with the fix.