Skip to content

feat: speed up exact filters - #3695

Open
knudtty wants to merge 3 commits into
claude/filter-keys-fetch-limitfrom
aaron/faster-exact-filter-values
Open

knudtty wants to merge 3 commits into
claude/filter-keys-fetch-limitfrom
aaron/faster-exact-filter-values

Conversation

@knudtty

@knudtty knudtty commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Speeds up exact filters immensely. Should help on both the search and for dashboard filters.

The old queries sent a query asking for many different individual map columns. ClickHouse's behavior is not set up well for that. Keep in mind, Maps are really stored as an array of key value pairs. ClickHouse will

  1. Load a row
  2. For the first map key column, walk the array until it finds the value. If value is not found, you just walked the whole array.
  3. Repeat for each map key column
  4. Move on to the next row

This leads to tons of extra reads, roughly one read per queried key per row. The new behavior is

  1. Load a row
  2. Filter for only specific keys you're looking for
  3. Move on to the next row

This means just one walk through the array per row.

Screenshots or video

Before

export-1794928260568-before.mp4

After

export-1794926363019-after.mp4

References

  • Linear Issue: Closes HDX-6053

@knudtty
knudtty requested a review from a team as a code owner November 17, 2026 15:24
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 17, 2026 4:11pm UTC
hyperdx-storybook Ready Ready Preview Nov 17, 2026 4:11pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 6fa2485

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 2
  • Production lines changed: 216 (+ 221 in test files, excluded from tier calculation)
  • Branch: aaron/faster-exact-filter-values
  • Author: knudtty

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

6 new clones introduced by this PR.

  • packages/api/src/models/source.ts:179-205 ↔ packages/api/src/models/source.ts:235-261 — 27 lines
  • packages/common-utils/src/core/metadata.ts:1345-1369 ↔ packages/common-utils/src/core/metadata.ts:1393-1414 — 25 lines
  • packages/common-utils/src/core/metadata.ts:1867-1876 ↔ packages/common-utils/src/core/metadata.ts:1996-2005 — 10 lines
  • packages/common-utils/src/core/metadata.ts:1876-1889 ↔ packages/common-utils/src/core/metadata.ts:2005-2019 — 14 lines
  • packages/common-utils/src/core/utils.ts:978-998 ↔ packages/common-utils/src/core/utils.ts:1026-1046 — 21 lines
  • packages/common-utils/src/types.ts:2769-2783 ↔ packages/common-utils/src/types.ts:2816-2830 — 15 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.17%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 1 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

  • 🟠 packages/common-utils/src/core/metadata.ts:490 — Compound expressions are spliced into the mapFilter IN list as if they were Map keys, which produces invalid SQL → parseKeyPath (metadata.ts:3718) only checks for the first [' and a trailing ']. A dashboard filter expression such as LogAttributes['b'] || LogAttributes['c'] therefore parses as column LogAttributes with key b'] || LogAttributes['c. That path[1] is then wrapped as '${path[1]}' and placed in k IN (...). If any other key on the same Map is in the call, so the column has at least two literals, the query is a syntax error and every filter in that group fails. Before this change the same expression worked as a plain groupUniqArray(expr). Fix: only group keys whose path[0] is a bare identifier and whose path[1] has no unescaped ', for example by requiring the whole key to match ^<ident>\['(?:[^'\\]|\\.)*'\]$. Leave everything else on the existing per-key path.
1 minor
  • 🔵 packages/app/src/hooks/useMetadata.tsx:144 — getMapColumn repeats the Map-column extraction already done in groupMapSubscripts → getMapColumn runs parseKeyPath and then unquoteIdentifier(path[0]) when path.length > 1. Metadata.groupMapSubscripts (packages/common-utils/src/core/metadata.ts:486-488) does the same thing inline. The two groupings must stay in sync: a fix to one, such as the compound-expression check above, would leave the other splitting keys differently. Export one helper (for example getMapColumnOfKey) next to parseKeyPath in core/metadata and use it in both places, as the repo's DRY rule requires.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: base accc11ea → PR head (aaron/faster-exact-filter-values), 6 files (2 production, 3 test, 1 changeset).
Intent: Speed up 'exact' filter-value lookups by (a) reading multiple keys of one Map column with a single groupUniqArrayMap(mapFilter(k IN (…))) aggregate instead of one groupUniqArray(Map['k']) per key, and (b) splitting client queries into one native-column group plus one per Map column, streamed in completion order.

✅ No critical issues found.

This is a focused, well-tested change. Both SQL paths (disableRowLimit true and the sampled CTE path) are unit-tested, including the mapParam/param alias wiring and faceted-key exclusion; an integration test exercises the real groupUniqArrayMap read against ClickHouse. Verified independently: per-key limit semantics are preserved by the -Map combinator, getColumn resolves through the cached getColumns so the per-column awaits in groupMapSubscripts add no real round-trips, facet consumers look up by .key so the new completion-order streaming does not break them, and inCompletionOrder surfaces the first rejection without orphaning unhandled rejections.

🟡 P2 -- recommended

  • packages/common-utils/src/core/metadata.ts:3398 -- The Map subscript literal is built by raw interpolation ('${path[1]}') of the parsed key segment and reused in both mapFilter(k IN (…)) and the [literal] lookup, so correctness depends on an undocumented contract that incoming key subscripts are already SQL-escaped (the b\'c test encodes this implicitly). A key whose subscript carries an unescaped quote would produce malformed SQL; the same raw-interpolation shape predates this PR for rendered keys, so this is a latent pattern, not a new exploit path.
    • Fix: Add an assertion or a comment at groupMapSubscripts stating that path[1] must arrive pre-escaped, or escape it explicitly before embedding so the filter and subscript share one escaping path.
🔵 P3 nitpicks (1)
  • packages/app/src/hooks/useMetadata.tsx:482 -- "Group keys by their Map column" is now implemented twice: client-side groupKeysByMapColumn/getMapColumn (to split queries) and server-side groupMapSubscripts (to share the SQL read), both deriving the column via parseKeyPath + unquoteIdentifier(path[0]). The two apply different rules (the server additionally verifies the column is a Map type and needs ≥2 keys), so they can drift as key-shape handling evolves.
    • Fix: Extract the shared "map column of a key" derivation into one helper imported by both layers, or add a cross-reference comment noting the two must stay consistent.

Reviewers (9): correctness, security, adversarial, performance, maintainability, testing, kieran-typescript, project-standards, previous-comments.

Testing gaps: none substantive — the diff adds unit coverage for both the direct and CTE SQL paths, faceted-key handling, special-character escaping, the native-columns-uncapped change, and completion-order error propagation, plus an integration test for the shared Map read.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 466 passed • 1 skipped • 1242s

Status Count
✅ Passed 466
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 6fa24855 Deployed Nov 17, 2026 by vercel[bot]
Preview – hyperdx-storybook — 6fa24855 Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant