Skip to content

feat(app): filter for or exclude a pattern from Event Patterns - #3665

Merged
SpencerTorres merged 1 commit into
alex/FR-385-compact-pattern-predicatefrom
alex/FR-385-pattern-filter-actions
Nov 18, 2026
Merged

SpencerTorres merged 1 commit into
alex/FR-385-compact-pattern-predicatefrom
alex/FR-385-pattern-filter-actions

Conversation

@alex-fedotyev

@alex-fedotyev alex-fedotyev commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Adds Filter for this pattern and Exclude this pattern to each row of Event Patterns. Each adds a search filter that ClickHouse evaluates over the whole time range, not only the 10,000 sampled rows, so the results table, histogram and counts follow it. Part of FR-385.

Stack

Layer 4 of 4 in a GitHub stack (stack #3682). GitHub shows each layer's base as the branch below it, but the stack's base is main: merging any PR lands it and every PR below it on main, one squashed commit each. Each PR's diff is only its own layer, and CI runs as if each targets main:

  1. feat(common-utils): share the Event Patterns drill-down predicate #3662 Share the pattern predicate in common-utils
  2. refactor(mcp): use the shared pattern predicate for drill-downs #3663 MCP uses the shared predicate
  3. feat(common-utils): add a compact form of the pattern predicate #3664 Compact form of the predicate, short enough for a URL
  4. feat(app): filter for or exclude a pattern from Event Patterns #3665 Filter for or exclude a pattern from Event Patterns (this PR)

Summary

  • Hovering a pattern row shows the two buttons in the row's hover strip, with the filter icons the app uses for other filter actions (IconFilter, IconFilterX). Filter for this pattern switches to the results table. Exclude this pattern stays on Event Patterns, which mines again without it.
  • The filter is a { type: 'sql' } condition built with the compact predicate from feat(common-utils): add a compact form of the pattern predicate #3664. It strips ANSI color codes first and splits on the same delimiters and whitespace as the browser miner (Python Drain3), so it selects the events the pattern describes, colored logs included.
  • Pattern filters show as Pattern pills after the other filters (= to include, != to exclude), within the same eight visible pills. A pill's ✕ removes it, and Clear all removes them with the rest.
  • They are kept in the URL and in saved searches like any other filter.
  • The pattern drawer is unchanged.

Filters are extended, not changed

A sidebar facet change rebuilds the filter list from facet state, which drops every SQL filter the facets can't represent. useSearchPageFilterState takes a new optional isPreservedFilter, and filters it matches are appended back after a facet update. Only the search page passes it, so other callers behave as before; a test pins that without the option a facet update still drops other SQL filters. With no pattern filters, every filter path emits what it does on main.

Guards

  • One include at a time. Filters are ANDed, so two includes would match nothing: a new include replaces the previous one. Excludes stack, and flipping a pattern between include and exclude replaces its filter.
  • A source change drops pattern filters, because the condition names the old source's body expression. They go through retainFiltersByColumns, so the existing "N filters didn't apply to this source" notice counts them.
  • Templates longer than 4,000 characters get no buttons, the same limit MCP uses.

Known limitations

  • The pattern drawer has no actions yet; it is due for a redesign.
  • The buttons live in the row's hover strip, like the results table's copy buttons, so they can't be reached from the keyboard yet. Keyboard access belongs with the drawer redesign.
  • Timestamps in a body that Drain keeps as literal tokens tie the pattern to that date and hour (2026 04 17T16 in the screenshots), so the filter misses events of the same shape from other hours.
  • Very generic templates such as <*> <*> match more events than the row suggests.
  • The filter strips CSI escape sequences (colors, cursor moves), the ones logs carry. The miner also strips rarer kinds, such as OSC hyperlinks, so a body with those doesn't match its pattern.
  • The compact form costs about 2x query time on exclusion filters (feat(common-utils): add a compact form of the pattern predicate #3664).
  • Pill text drops punctuation, because it shows the template.
  • Dashboard pattern tiles don't get the buttons.
  • UI and MCP still mine with different delimiters, so they produce different templates for the same logs, even though both filters now come from one function.

Test plan

  • unit: searchFilters.test.ts (preserved filters survive a facet update; clear all and a source change drop them; without the option a facet update still drops other SQL filters), ActiveFilterPills.test.tsx (a pattern pill renders and its ✕ calls the handler; pattern pills count toward the eight visible pills), patternFilter.test.ts (build and parse, including a quoted value expression; run on bodies the way ClickHouse does, including color codes and Python-only whitespace; only filters it built read back; no filter for a template with punctuation inside a token; flips both ways, one include across columns, excludes stack), PatternRowActions.test.tsx (both buttons build the full filter; none for a template too long to filter on)
  • e2e: event-patterns.spec.ts "pattern filters", 3 tests: an exclude from a row survives a facet change; Filter for this pattern opens the results table with a Pattern = pill; a source change drops the filter with the notice. CI runs them on this PR. I also replayed the three flows with the same selectors on a dev stack against the OpenTelemetry demo logs:
    • exclude from a row: 347,939 to 312,989 events, the pattern leaves the list, the drawer stays closed
    • SeverityText = info keeps the exclusion, and a second exclusion stacks (244,332)
    • Filter for this pattern: results table, 22,244 events, no error
    • switching to Demo Traces: "4 filters didn't apply to this source and were removed", no pattern pills left
  • manual, on the same changes before they were split into this stack: 8 stacked exclusions (318K to 178K events), a saved search restores all 8 pills, a reload keeps them
  • manual, after the review fixes: an exclusion with the ANSI step runs (343,228 to 310,958 events, no error); seven ServiceName values and two exclusions show eight pills and "+1 more", which expands to both pattern pills. In ClickHouse, a colored body fits its template only with the ANSI step, and 8 real exclusions leave the same 177,130 events in about the same time with or without it
  • ClickHouse: the ANSI step and the Python whitespace grow the URL for 8 exclusions from 11,010 to 12,786 characters
  • states. Loading: as with any filter change, the previous list stays on screen until the new sample comes back (about 1 s after a facet change, 2.5 to 3 s after an exclusion here, since the regex is slower), then the table's "Loading results" footer shows while it mines; the new pill shows right away. Empty: a time range without events shows "No results found" and no buttons. Error: a pattern filter that ClickHouse rejects (a hand-edited URL naming a missing column) shows the existing query error, and removing its pill recovers
  • viewport: at 1440 x 900 and 1280 x 800 the buttons fit in the row. At 1024 x 768 the results pane is wider than the window, so these buttons and the results table's existing copy buttons both sit past its right edge
  • light and dark mode, screenshots below
  • tsc --noEmit; eslint with no warnings on added lines; the 74 app suites related to the changed files: 71 pass (1,390 tests), and the other 3 (SearchWhereInput, DBSearchPageAlertModal, DBEditTimeChartForm) fail the same 43 tests on main in my environment (TypeError reading addEventListener while rendering Mantine notifications); knip; yarn dupes

PatternRowActions has no story: it is two existing DBRowTableIconButtons inside the table's hover strip, covered by a unit test and the e2e tests above.

Screenshots

Row buttons on hover, light and dark:

light row actions

dark row actions

Two exclusions next to a facet filter, with the list mined again without them, light and dark:

light pills

dark pills

After Filter for this pattern on top of those, light:

light include

@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 18, 2026 1:46am UTC
hyperdx-storybook Ready Ready Preview Nov 18, 2026 1:46am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 132ef49

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Minor
@hyperdx/api Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

✅ No issues found.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. No P0/P1 ship-blockers. The change is well-scoped, strongly tested, and prior inline/round-1 review feedback is all reflected in the current code. Security (SQL-injection surface via the template-derived predicate), filter double-emission, withPatternFilter dedup, and preservedFiltersRef timing were each verified sound. The items below are recommended follow-ups and nits.

🟡 P2 — recommended

  • packages/app/src/DBSearchPage.tsx:1490 — The pattern-pill removal path (removePatternFilter, which drops filters by condition equality) has no unit or e2e coverage; the pill test only asserts the handler is invoked, and no e2e clicks a pattern pill's ✕.
    • Fix: Add a test that removePatternFilter drops only the filter whose condition matches and leaves facet/other pattern filters intact, plus a no-op case when the condition is absent.
    • testing, adversarial
  • packages/app/src/components/Patterns/patternFilter.ts:48 — Each stacked exclusion appends a full-time-range NOT match(...) whose inner replaceRegexpAll(replaceRegexpAll(body, ANSI, ''), sep, ' ') normalization is byte-identical across every predicate, so without ClickHouse common-subexpression elimination N exclusions run ~2N whole-column regex rewrites plus N matches per row over the entire dataset.
    • Fix: Confirm whether ClickHouse de-duplicates the identical normalized-body subexpression across ANDed terms; if not, hoist the normalized body into one shared/aliased expression so only the per-pattern match() regex scales with N.
    • performance
🔵 P3 nitpicks (6)
  • packages/app/src/components/ActiveFilterPills.tsx:281 — When facet pills alone reach MAX_VISIBLE_PILLS (8), visiblePatternFilters slices to zero in collapsed mode, so a just-added pattern filter renders no pill and can only be reached by expanding +N more.
    • Fix: Reserve at least one visible slot for pattern pills, or render pattern pills ahead of truncating facet pills, so a newly added exclusion is always visible.
    • adversarial, correctness, testing, kieran-typescript
  • packages/app/src/components/Patterns/patternFilter.ts:28 — The filter strips only ANSI CSI sequences while the miner strips all ANSI (strip-ansi), so bodies carrying non-CSI escapes (e.g. OSC hyperlinks) are clustered into a pattern but miss its include/exclude filter.
    • Fix: Either broaden the SQL strip to the same escape classes strip-ansi removes, or keep the documented CSI-only scope and surface the known gap to users.
    • correctness
  • packages/app/src/hooks/usePatterns.tsx:311 — Pattern.template is declared string | undefined but assigned from any sources (clusterTemplates[patternId] / __hdx_pattern), so a non-string runtime value can land in a field typed as string; only PatternTable's typeof guard currently protects consumers.
    • Fix: Coerce or guard the value to string | undefined at assignment so the field's type reflects reality.
    • kieran-typescript
  • packages/app/src/components/ActiveFilterPills.tsx:308 — Two filters with an identical condition (reachable via a hand-edited URL or saved search) produce duplicate key={filter.condition} pills, colliding on the React key; removal then strips both. (Removing both is intentional, but the key collision is not.)
    • Fix: Key the pattern pill on index plus condition and de-duplicate patternFilters by condition before render.
    • adversarial
  • packages/app/src/components/PatternTable.tsx:105 — The renderRowActions guards (onPatternFilter == null → no actions, and non-string row.template → null) are untested; PatternTable has no test file.
    • Fix: Add a render test for a row with no string template and for the no-handler case.
    • testing
  • packages/app/src/components/Patterns/PatternRowActions.tsx:43 — The new "Exclude this pattern" action has no agent-accessible equivalent: the MCP tool surfaces only the combined whereSnippet (table AND user AND predicate), so an agent cannot compose AND NOT predicate without re-deriving the server-side predicate, breaking agent/user parity for the exclude path.
    • Fix: Surface the bare per-pattern predicate (e.g. a patternPredicate field) or document negation in the tool usage text so an agent can reproduce the exclusion.
    • agent-native

Reviewers (12): correctness, security, adversarial, testing, maintainability, project-standards, performance, kieran-typescript, julik-frontend-races, previous-comments, agent-native, learnings-researcher.

Testing gaps:

  • No coverage of the pattern-pill removal path (removePatternFilter by-condition) in unit or e2e.
  • No ActiveFilterPills case where facet pills alone ≥ 8 with pattern filters present (collapsed reachability + hiddenCount).
  • No PatternTable test for the renderRowActions guards (null handler, non-string template).
  • fits helper in patternFilter.test.ts re-implements ClickHouse replaceRegexpAll/match semantics in JS, so a divergence from real ClickHouse RE2 behavior would go uncaught by unit tests.
  • No test for bodies with non-CSI ANSI escapes, nor for an empty/whitespace template passing the PatternRowActions string guard.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@alex-fedotyev

Copy link
Copy Markdown
Contributor Author

Review findings outside the inline threads:

  • ANSI codes (major): fixed in 819afc5. The filter strips CSI sequences before matching, so colored logs match their pattern, and the parser only accepts filters built that way. In ClickHouse a colored body fits its template only with that step, and 8 real exclusions leave the same events in about the same time with or without it. Rarer escape kinds the miner also strips, such as OSC hyperlinks, are listed under known limitations.
  • The exclude assertion now checks the full filter: fixed in 819afc5.
  • A pattern without a template gets no buttons instead of a throw: fixed in 819afc5.
  • removePatternFilter removes every filter with the pill's condition on purpose: an identical condition is the same pattern filter (parsePatternFilter accepts it), and keeping a copy would leave the pattern applied after its pill is gone.

isPreservedFilter: isPatternFilter,
});

const patternFilters = useMemo(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Two separate derivations of the pattern-filter set

useSearchPageFilterState already splits pattern filters out of filters through isPreservedFilter (searchFilters.tsx:129). DBSearchPage then parses filters again for the pills and adds its own remove/add handlers inline. Have the hook return the preserved filters and a remover, or move this into a usePatternFilters hook in patternFilter.ts, so the pills and the facet-preservation logic use one source.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

searchFilters.tsx stays pattern-agnostic on purpose: it only knows isPreservedFilter, and the page derives the pills from the same filters the hook reads, so the two always agree.

@alex-fedotyev

Copy link
Copy Markdown
Contributor Author

Second-round findings outside the inline threads:

  • Keyboard access: the buttons reuse the table's hover strip, like the results table's copy buttons (tabIndex={-1}, shown on hover). Making the strip keyboard-reachable changes every table, so it belongs with the drawer redesign; it's listed under known limitations.
  • Round trip with quotes (P2): fixed in 26c9359, with a quoted value expression. A template with punctuation inside a token gets no filter, and the miner never produces one.
  • preservedFiltersRef timing (P2): same as knownColumnsRef next to it. The ref catches up on every commit, and facet and pattern changes come from separate clicks, so they never share a flush.
  • Flips both ways and an include on another column: tested in 26c9359.
  • What isPreservedFilter may match: documented in 26c9359.
  • CSI-only stripping: listed under known limitations.
  • Parsing the ANSI wrapper with startsWith/endsWith: the round trip is pinned by tests, so a change in its shape fails them.
  • keepPreserved on by default: only the search page passes isPreservedFilter, and it wants preserved filters kept on every update except clear all and a source change, the two calls that opt out.

That's the second round of automated findings, so per AGENTS.md I'm leaving any further ones for review.

valueExpression: string;
onFilter: (filter: ConditionFilter, action: PatternFilterAction) => void;
}) {
const filters = useMemo(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Each row builds the same compact predicate twice, and the exclude == null check can never fail on its own

buildPatternFilter returns null only when buildCompactPatternPredicate returns null, and include and exclude are built from that same predicate. Build the include filter once and derive the exclude from it, for example with an invertPatternFilter helper in patternFilter.ts that adds the NOT prefix. Then drop the separate exclude == null check. This halves the regex work on every pattern row mount.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

isPreservedFilter: isPatternFilter,
});

const patternFilters = useMemo(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Pattern-filter state (parse list, add, remove) is hand-written inside DBSearchPage, which also tracks the same filters a second way

useSearchPageFilterState already collects the same set through isPreservedFilter into preservedFiltersRef, so these filters now have two copies that must stay in sync. Move patternFilters, addPatternFilter and removePatternFilter into a usePatternFilters(filters, handleSetFilters) hook in components/Patterns/patternFilter.ts, or expose the preserved filters from useSearchPageFilterState. That keeps pattern-filter handling in one module instead of the multi-thousand-line page.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 337 production lines changed (Tier 2 max: < 250)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 10
  • Production lines changed: 337 (+ 385 in test files, excluded from tier calculation)
  • Branch: alex/FR-385-pattern-filter-actions
  • Author: alex-fedotyev

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 106 in total (1.14%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 472 passed • 1 skipped • 1030s

Status Count
✅ Passed 472
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@alex-fedotyev
alex-fedotyev force-pushed the alex/FR-385-pattern-filter-actions branch from 7695e2c to d1f603c Compare November 16, 2026 18:27
@alex-fedotyev
alex-fedotyev changed the base branch from main to alex/FR-385-compact-pattern-predicate November 16, 2026 18:27
@alex-fedotyev

Copy link
Copy Markdown
Contributor Author

Rebuilt this as a GitHub stack with one commit per layer; the content is unchanged, and the force-push is in the timeline. The "Fixed in" commits in the replies above are from before the rebuild.

Event Patterns groups a 10,000-row sample into patterns, but a pattern
could not be acted on: there was no way to see every event that fits it,
or to hide it and look at what is left.

Hovering a pattern row now shows two buttons. "Filter for this pattern"
adds a filter for the events that fit the pattern and switches to the
results table. "Exclude this pattern" adds the negated filter and stays on
Event Patterns, which mines again without it. The filter is a SQL condition
built with the shared compact pattern predicate, so it runs in ClickHouse
over the whole time range, and the histogram and counts follow it.

Pattern filters show as pills next to the other filters and are kept in
the URL and in saved searches like any other filter. A sidebar facet change
rebuilds the filter list from facet state, which would drop them, so
useSearchPageFilterState takes an isPreservedFilter option that keeps
matching filters through facet changes. Only the search page passes it;
other callers behave as before, and a test pins that. "Clear all" still
removes them.

Two guards keep the workflow out of broken states. A new include replaces
the previous one, because two includes AND together and match nothing. A
source change drops pattern filters through the existing "filters didn't
apply" notice, because the condition names the old source's body
expression.

Patterns come from Python Drain3 after strip-ansi, so the filter strips
ANSI CSI sequences (colors) first and splits on Drain3's whitespace;
colored logs match their pattern. Pattern pills count toward the eight
visible pills, after the facet pills.

Part of FR-385.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@SpencerTorres SpencerTorres left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No major issues across the whole stack. Thanks for adding this 🚀

@SpencerTorres
SpencerTorres merged commit ca4855d into main Nov 18, 2026
29 checks passed

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 132ef491 Deployed Nov 18, 2026 by vercel[bot]
Preview – hyperdx-storybook — 132ef491 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants