Skip to content

feat: create SQL query dashboard filters and fill their dropdowns - #3678

Merged
karl-power merged 8 commits into
mainfrom
sql-dashboard-variables-app
Nov 18, 2026
Merged

karl-power merged 8 commits into
mainfrom
sql-dashboard-variables-app

Conversation

@karl-power

@karl-power karl-power commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the "SQL query values" dashboard filter type to the UI: users can create the filter, and its dropdown lists the first column of a ClickHouse query. This is the last PR in the stack; earlier PRs added the schema (#3672) and API/MCP support (#3674).

  • renderSqlQueryFilterValuesQuery in common-utils expands time macros and other filters' variables, wraps the SQL as SELECT toString(tupleElement(tuple(*), 1)) AS value FROM (...) LIMIT 10000, and strips a trailing ;. Time params are sent only when the SQL uses them.
  • useSqlQueryFilterValues runs each filter's query on its connection with max_execution_time: 30. Nulls, empty strings and duplicates are dropped, and the query's order is kept, so the dropdown doesn't sort these values.
  • useDashboardFilterValues merges these values and errors with the other filter types. A query error shows as the dropdown's caution, as do variables the query references that have no selection yet.
  • The editor form has a connection picker, a SQL editor with time macro and variable autocomplete ($__sourceTable and $__filters are left out, since the filter has no source), and the variable name. The filter is variable-only, so there are no broadcast or variable toggles.
  • Preview values (or Cmd+Enter) runs the query over the dashboard's time range and lists up to 20 values, each truncated to 20 characters with the full value on hover, plus "and N more".

Screenshots or video

Untitled.mov

How to reproduce

On main:

  1. Open a dashboard, click Edit filters and variables, then Add new filter.
  2. The Type picker has no "SQL query values" option.

On this branch:

  1. Open the same picker and choose SQL query values.
  2. Set Display name to Service, pick the demo connection, and enter:
    SELECT DISTINCT ServiceName FROM default.otel_logs
    WHERE $__timeFilter(Timestamp)
    ORDER BY 1
  3. Click Preview values. The preview lists service names.
  4. Set Variable name to svc, click Save filter, and close the modal. The Service dropdown lists the same services in the query's order.
  5. Add a Static list filter Severity with options INFO and ERROR and variable sev. Change the SQL to add AND $__filter(SeverityText, $sev). Picking a severity narrows the Service dropdown.
  6. Change the SQL to SELECT nope FROM default.otel_logs and preview. The preview shows the ClickHouse error, and after saving, the dropdown shows it as a caution.

References

@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 15:49
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 18, 2026 2:17pm UTC
hyperdx-storybook Ready Ready Preview Nov 18, 2026 2:17pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: cffcab2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@karl-power
karl-power added this pull request to stack #3675 November 16, 2026 15:49
@karl-power
karl-power force-pushed the sql-dashboard-variables-app branch from cf6ccc5 to d411792 Compare November 16, 2026 15:50
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/app/src/hooks/usePromqlLabelFilterValues.tsx:127-150 ↔ packages/app/src/hooks/useSqlQueryFilterValues.tsx:109-132 — 24 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 107 in total (1.14%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 476 passed • 1 skipped • 1028s

Status Count
✅ Passed 476
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 639 production lines changed (Tier 2 max: < 250)
  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 11
  • Production lines changed: 639 (+ 674 in test files, excluded from tier calculation)
  • Branch: sql-dashboard-variables-app
  • Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 P0/P1 · 🟡 2 P2 · 🔵 0 P3

2 posted as inline comment(s) on the changed lines.


Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. No P0/P1 ship-blockers. The feature reuses the established filter-value hook pattern cleanly, prior review feedback is verified as addressed, variable interpolation reuses the existing escapeSqlString path with no new injection surface, and agent/MCP parity is covered by the earlier stacked PRs. One P2 and a handful of P3 nits below.

🟡 P2 — recommended

  • packages/common-utils/src/filters.ts:933 — The dropdown query wraps user SQL as SELECT DISTINCT toString(tupleElement(tuple(*), 1)) AS value FROM (...) LIMIT 10000 with no outer ORDER BY, so ClickHouse's parallel DISTINCT does not preserve the inner query's ordering, and when the inner query yields more than 10,000 distinct values the un-ordered LIMIT returns an arbitrary subset that can shift between refetches — contradicting the "values in query order" guarantee that DashboardFilters.tsx relies on via sort={false}.
    • Fix: Carry an explicit row-order key through the subquery and apply an outer ORDER BY so both the ordering and the capped subset are deterministic, or downgrade the ordering guarantee to best-effort in the docs and UI.
    • correctness, adversarial
🔵 P3 nitpicks (5)
  • packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:106 — runPreview calls invalidateQueries({ queryKey: [SQL_QUERY_FILTER_VALUES_QUERY_KEY, resolvedConnection] }), a non-exact prefix match that also invalidates every live dashboard SQL-query-filter dropdown and any other preview sharing that connection, forcing needless refetches (masked by keepPreviousData, so no wrong values).
    • Fix: Scope invalidation to the preview's exact expanded sql/params or give preview queries a distinguishing key segment.
    • kieran-typescript, julik-frontend-races, adversarial
  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:37 — toFilterValues filters rows with !!value, which drops legitimate empty-string first-column values (e.g. an empty tag/env) in addition to nulls, so they can never appear in or be selected from the dropdown.
    • Fix: Narrow with value != null instead of !!value, or document that empty strings are intentionally excluded.
    • kieran-typescript, adversarial
  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:103 — timeout_overflow_mode: 'break' plus LIMIT 10000 return partial result sets as a successful response with no truncation signal, so a timed-out or capped query shows an incomplete value list that looks complete.
    • Fix: Detect truncation (row count equals the limit, or response statistics) and surface a "results may be incomplete" indicator.
  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:89 — The retry guard !(error instanceof ClickHouseQueryError) && failureCount < 1 likely never retries real transport failures, since the ClickHouse client appears to wrap network errors as ClickHouseQueryError, making the intended one-retry-on-transport-error path effectively dead (the test exercises a plain Error, not the wrapped form).
    • Fix: Distinguish transient transport failures from query errors via the error cause or HTTP status rather than instanceof ClickHouseQueryError alone, and cover the wrapped-transport-error case in a test.
  • packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:102 — runPreview only re-runs via invalidateQueries when previewed equals the current connection+template; previewing query A, then B, then editing back to A and clicking Preview skips the invalidate, so react-query serves A's still-fresh cached values (staleTime 5 min) without actually re-running.
    • Fix: Trigger the refetch unconditionally when canPreview, or use staleTime: 0 for the preview query, if every click must re-run.

Reviewers (13): correctness, testing, maintainability, project-standards, agent-native, learnings-researcher, kieran-typescript, julik-frontend-races, performance, reliability, security, adversarial, previous-comments.

Testing gaps:

  • No E2E coverage exercising the modal → ClickHouse → dropdown flow end-to-end (acknowledged follow-up by the author).
  • useDashboardFilterValues only tests the happy SQL-value merge; propagation of SQL erroredFilterIds/filterErrorMessages/isError into the combined result is unasserted.
  • The empty-values ("No values") preview branch and the new DashboardFilters.tsx SQL-filter branches (sort suppression, pending-variable wiring) are untested.
  • No test confirms a malicious sibling-filter value cannot break out of escapeSqlString when interpolated into a SQL-query template, nor that empty-string / ordering / >10k-value behaviors hold.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@karl-power

Copy link
Copy Markdown
Contributor Author

Addressed the findings outside the changed lines in e9c4dca:

  • Interval macros: expandSqlQueryFilterTemplate now rejects $__timeInterval, $__timeInterval_ms and $__interval_s, and the editor no longer suggests them.
  • Preview re-run: the preview shows the loader while isFetching, so the previous query's values aren't shown as the new results.
  • Stale comment in DashboardFilters.tsx: updated to cover SQL query filters.

The jscpd clone with usePromqlLabelFilterValues is left as is; sharing it would mean a helper across both hooks, which is outside this PR's scope.

],
// Keep the last values while a new range or selection loads, rather than
// blanking the dropdown.
placeholderData: keepPreviousData,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — keepPreviousData has no effect inside useQueries when the key changes, so the dropdown goes blank on every reload

In react-query v5, useQueries matches observers by query hash. When the expanded SQL or the time params change, the query gets a new observer that has no previous data, so keepPreviousData returns undefined and values drops to [] until the fetch finishes. This contradicts the comment on the line above it. Follow the sibling hooks (usePromqlLabelFilterValues.tsx:102, useQueriedDashboardFilterValues.tsx:373): build a key prefix of ['dashboard-filter-sql-query-values', call.connection], then in placeholderData return the most recently updated cached entry from queryClient.getQueriesData({ queryKey: prefix }). This filter's own variables can change the SQL, so the prefix probably also needs the filter id.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

// Only the params the SQL reads, so a query that ignores the time range
// keeps its cache key, and isn't re-run, when the range changes.
params: Object.fromEntries(
Object.entries(timeParams).filter(([name]) => sql.includes(`{${name}:`)),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — A raw {intervalSeconds:Int64} / {intervalMilliseconds:Int64} param passes validation but is never sent, so the query fails at runtime

expandSqlQueryFilterTemplate rejects the interval macros (line 888), but the params only include start/end. A template that writes {intervalSeconds:Int64} directly, as TIME_CHART_EXAMPLE_SQL in rawSqlParams.ts shows, passes validateSqlQueryFilterTemplate in the form, API and MCP. Every run then fails in ClickHouse with a missing-substitution error. Reject templates that reference {intervalSeconds: or {intervalMilliseconds: in expandSqlQueryFilterTemplate, using the same "have no time buckets" message, so saving fails up front.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of thoughts, looks pretty good though and works well! One other thing that could be nice is a couple of E2E exercising this filter type

Comment thread packages/common-utils/src/filters.ts Resolved
Comment thread packages/common-utils/src/filters.ts Resolved

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for adding the new connection variable support!

Add a "SQL query values" type to the dashboard filter editor and populate
the filter's dropdown from its query.

- Render the filter's query in common-utils: expand time macros and other
  filters' variables, select the first column as a string, and cap the
  result at 10,000 rows
- Run each SQL query filter's query on its connection and merge the values
  and errors into the dashboard's filter values
- Keep the query's row order in the dropdown and show a caution while a
  referenced variable has no selection
- Add the editor form: connection picker, SQL editor with macro and
  variable autocomplete, variable name, and an on-demand preview of the
  values over the dashboard's time range

Refs HDX-5524
- Reject interval macros in SQL query filters, which have no time buckets
- Validate the SQL on save against the dashboard's other variables
- Build the editor's unsupported-macro list from the shared constants
- Show the preview loader while a re-run fetches
],
// Keep the last values while a new range or selection loads, rather than
// blanking the dropdown.
placeholderData: keepPreviousData,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — placeholderData: keepPreviousData does nothing inside useQueries, so the dropdown blanks on every range or selection change

In TanStack Query v5, useQueries creates a new QueryObserver whenever a query's key changes, so keepPreviousData never has previous data to keep. A filter whose SQL uses $__timeFilter, or references another filter's variable, drops to a loader on each change, contrary to the comment. Use the prefix-based placeholderData lookup (queryClient.getQueriesData({ queryKey: [SQL_QUERY_FILTER_VALUES_QUERY_KEY, call.connection] }), newest by dataUpdatedAt) that usePromqlLabelFilterValues.tsx and useQueriedDashboardFilterValues.tsx already use.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

previewed?.connection === resolvedConnection &&
previewed.sqlTemplate === sqlTemplate
) {
void queryClient.invalidateQueries({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — Preview re-run invalidates every SQL query filter on the connection, and the first preview can return cached dashboard results

invalidateQueries({ queryKey: [KEY, resolvedConnection] }) is a prefix match, so each repeat click also refetches every SQL query filter the dashboard behind the modal shows on that connection. When editing an existing filter, the first click shows results the dashboard cached in the last 5 minutes (staleTime) instead of running the query. Re-run only the preview's own query on every click: return refetch from the hook, or invalidate the exact key including the SQL and params. The test only checks that invalidateQueries was called with a mocked key, so also assert that the query runs again.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@karl-power
karl-power merged commit 3acd5ba into main Nov 18, 2026
27 of 28 checks passed

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — cffcab26 Deployed Nov 18, 2026 by vercel[bot]
Preview – hyperdx-storybook — cffcab26 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants