Repository navigation
feat(common-utils): add a compact form of the pattern predicate - #3664
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: ee7d365 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| return `match(${normalized}, '${escapeSqlString(compactRegex(template))}')`; | ||
| } | ||
|
|
||
| const unescapeSql = (literal: string) => |
There was a problem hiding this comment.
🔵 minor — The inverse of escapeSqlString now exists in three places
unescapeSql is new here, a copy of it is pasted into __tests__/pattern-predicate.test.ts:49, and filters.ts:190 (getBooleanOrUnquotedString) already does the same unescape inline. Add an exported unescapeSqlString next to escapeSqlString in core/utils.ts and use it in all three places, so the escape and unescape rules live in one file.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Hoisting an unescapeSqlString next to escapeSqlString and switching filters.ts over would touch files outside this PR, so that's left for a separate cleanup.
PR Review
1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor 1 posted as inline comment(s) on the changed lines. Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
Deep ReviewScope: ✅ No critical issues found. The equivalence logic is sound for all realistically-mined templates, SQL escaping is applied on every template-derived path, and the round-trip guard ( 🟡 P2 -- recommended
🔵 P3 nitpicks (3)
Reviewers (9): correctness, adversarial, security, testing, maintainability, kieran-typescript, api-contract, project-standards, previous-comments. Testing gaps:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
|
Review findings outside the inline threads, fixed in e6804d4:
|
| return `match(${normalized}, '${escapeSqlString(compactRegex(template))}')`; | ||
| } | ||
|
|
||
| const unescapeSql = (literal: string) => |
There was a problem hiding this comment.
🔵 minor — The inverse of escapeSqlString is now written out by hand in three places
unescapeSql here, its copy in pattern-predicate.test.ts:31, and the inline .replace(/\\\\/g, '\\').replace(/''/g, "'") in filters.ts:190 all undo escapeSqlString (core/utils.ts:147). Fix: export one unescapeSqlString next to escapeSqlString in core/utils.ts and import it in all three places. The pattern for a quoted SQL literal, '((?:[^'\\]|\\\\|'')*)', now appears three times in this file (lines 150 twice, 189) and could become one shared constant.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Same answer as the earlier thread: hoisting the helper means touching core/utils.ts and filters.ts, outside this PR, and building the two parser regexes from a shared string would make them harder to read than the literals.
|
Second-round findings outside the inline threads:
That's the second round of automated findings, so per AGENTS.md I'm leaving any further ones for review. |
| return `match(${normalized}, '${escapeSqlString(compactRegex(template, options))}')`; | ||
| } | ||
|
|
||
| const unescapeSql = (literal: string) => |
There was a problem hiding this comment.
🔵 minor — unescapeSql is a third hand-written inverse of escapeSqlString, and the test file copies it again
Export an unescapeSqlString from packages/common-utils/src/core/utils.ts, next to escapeSqlString. Then use it here, in __tests__/pattern-predicate.test.ts:31, and in filters.ts:190 (getBooleanOrUnquotedString), which already does the same \\→\ and ''→' replacements inline.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
🔵 Tier 2 — Low RiskSmall, isolated change with no API route or data model modifications. Why this tier:
Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
E2E Test Results✅ All tests passed • 470 passed • 1 skipped • 1001s
Tests ran across 4 shards in parallel. |
b6f789e to
aefab4f
Compare
|
Rebuilt this as a GitHub stack with one commit per layer; the content is unchanged, and the force-push is in the timeline. The "Fixed in" commits in the replies above are from before the rebuild. |
aefab4f to
7dfdfc1
Compare
buildPatternPredicate repeats the separator class between every token. With all ASCII punctuation as delimiters that is about 30 characters per token. The app keeps pattern filters in the URL, and with this form two exclusions of long patterns already pass Node's default 16 KB header limit. buildCompactPatternPredicate builds the same match in a different shape: replaceRegexpAll() turns every separator run in the value into one space, and the regex matches tokens joined by single spaces, so the separator class appears once. On 8 templates mined from the demo logs, 8 exclusions take 11K characters in the URL instead of 104K, and each template matches the same number of events in ClickHouse with either form. The cost is speed: those 8 exclusions over 15 minutes of logs took 0.95 s instead of 0.45 s. It returns null for a template whose token holds one of the extra delimiters, where the two forms would disagree. parsePatternPredicate reads both forms. Part of FR-385. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7dfdfc1 to
ee7d365
Compare
Adds
buildCompactPatternPredicate: the same match asbuildPatternPredicatein much shorter SQL, for pattern filters that live in the URL (#3665).parsePatternPredicatereads both forms. Part of FR-385.Stack
Layer 3 of 4 in a GitHub stack (stack #3682). GitHub shows each layer's base as the branch below it, but the stack's base is
main: merging any PR lands it and every PR below it onmain, one squashed commit each. Each PR's diff is only its own layer, and CI runs as if each targetsmain:Summary
buildPatternPredicaterepeats the separator class between every token. With all ASCII punctuation as delimiters that is about 30 characters per token, before SQL escaping and URL encoding. The compact form turns every separator run in the value into one space, then matches tokens joined by single spaces, so the class appears once. ForGET api v1 users <*>:Why a second form
Measured on the OpenTelemetry demo logs, with 8 templates the app mined from 15 minutes of logs:
buildPatternPredicatebuildCompactPatternPredicateEach template also matched the same number of events with either form (15,198 to 30,056). Node's default header limit is 16 KB, so with the full form two exclusions of long patterns already break the page URL, while the compact form fits about ten. The price is about 2x on the exclusion filters. Keeping the compact form in the URL and querying with the full form would win that back; that is left for a follow-up.
Test plan
replaceRegexpAll, thenmatchparsePatternPredicateunder every option set; a hand-written condition is rejected both by the separator check and by the round tripyarn ci:lintandyarn ci:unitinpackages/common-utils(47 suites, 3,160 tests), knip,yarn dupes(no new duplicate code)