Skip to content

feat: add data source variables to dashboard filters and tiles - #3654

Closed
pulpdrew wants to merge 4 commits into
drew/source-vars-2from
drew/source-vars-3
Closed

pulpdrew wants to merge 4 commits into
drew/source-vars-2from
drew/source-vars-3

Conversation

@pulpdrew

@pulpdrew pulpdrew commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

PR 3/5 adding data source and data connection type variables. This PR adds the UI for defining and rendering a source variable / filter. The dashboard page substitutes source variables for source IDs when rendering the dashboard, and shows placeholder / error states for invalid or missing source variables.

The variable cannot yet be referenced by a tile or another filter via the UI but can via MCP, API, or Import.

Source and connection variables allow for dashboards variable inputs that define a source or connection queried by a tile or other filters. Selected values are stored in source or connection id slots in the saved chart config, in the $varName format, then converted to source or connection IDs at runtime when there is a selection.

Variables are substituted prior to rendering the dashboard, at the top level, so that components and configs down the stack can just use source and connection IDs like they have been.

Direct $source references are not yet allowed in SQL tiles - the source is selected at the chart level and $__sourceTable can be used to reference it. Connection variables are mostly only useful for (a) raw sql tiles and (b) source variables that can be scoped to a single connection, dynamically.

Screenshots or video

Screenshot 2026-11-16 at 12 28 10 PM Screenshot 2026-11-16 at 12 28 36 PM

How to test on Vercel preview

The E2E feature can be tested on PR 5: #3656

References

@pulpdrew
pulpdrew added this pull request to stack #3653 November 13, 2026 20:02
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 18, 2026 12:25am UTC
hyperdx-storybook Ready Ready Preview Nov 18, 2026 12:25am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 1017e51

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Minor
@hyperdx/api Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 467 passed • 1 skipped • 1036s

Status Count
✅ Passed 467
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

4 new clones introduced by this PR.

  • packages/api/src/mcp/tools/dashboards/queryTile.ts:69-87 ↔ packages/api/src/mcp/tools/dashboards/queryTiles.ts:175-193 — 19 lines
  • packages/api/src/mcp/tools/trace/breakdown.ts:73-89 ↔ packages/api/src/mcp/tools/trace/waterfall.ts:339-354 — 17 lines
  • packages/api/src/utils/zod.ts:572-582 ↔ packages/api/src/utils/zod.ts:583-603 — 11 lines
  • packages/app/src/components/SourceMultiSelect.tsx:44-63 ↔ packages/app/src/components/SourceSelect.tsx:194-213 — 20 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 106 in total (1.14%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@pulpdrew
pulpdrew marked this pull request as ready for review November 16, 2026 20:36
@pulpdrew
pulpdrew requested a review from a team as a code owner November 16, 2026 20:36
@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 845 production lines changed (Tier 2 max: < 250)
  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 12
  • Production lines changed: 845 (+ 674 in test files, excluded from tier calculation)
  • Branch: drew/source-vars-3
  • Author: pulpdrew

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

Comment thread packages/app/src/DashboardFilters.tsx Resolved
Comment thread packages/app/src/DashboardFilters.tsx Resolved
Comment thread packages/app/src/DBDashboardPage.tsx Resolved
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

5 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 4 minor

5 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: PR #3654 — data source / connection-type dashboard variables ($varName references stored in source/connection id slots, resolved to IDs before rendering, with placeholder/error states). Diff reviewed against base ef79c916, excluding ce-plugin/.

✅ No critical issues found. The change is well-structured and well-tested: reference resolution is centralized in packages/common-utils/src/referenceVariables.ts, the happy paths and the global-requirement blocking path have e2e coverage, and resolveFilterReferences (including the new incompatible-kind branch) has unit coverage. The prior substantive review comment (@jordan-simonovski: share one rule via isSourceOfferedByVariable) is addressed — the picker and useDashboardFilters now both call isSourceOfferedByVariable.

🟡 P2 -- recommended

  • packages/app/src/DBDashboardPage.tsx:2538 -- on tile save, restoreConfigReferences(newChart.config, chart.config) re-applies the stored $var source/connection fields onto the edited config, so a source or connection change made in the editor for a variable-referencing raw-SQL tile could be silently discarded.
    • Fix: confirm the tile editor cannot alter a reference-backed field, or restore the reference only when that field was left unchanged by the edit.
  • packages/app/src/DBDashboardPage.tsx:1297 -- the new tile placeholder branches isSourceOnOtherConnection and isSourceKindUnsupported (the latter with data-testid="tile-unsupported-source-kind") have no accompanying test exercising them.
    • Fix: add a unit or e2e test that drives a variable selection into each unusable state and asserts the placeholder renders.
  • packages/app/src/DashboardFilters.tsx:304 -- the incompatible-variable error rendering in the filter bar (and the mirrored path in AdhocDashboardFilter.tsx) is new behavior without a component-level test; only the underlying resolveFilterReferences util is covered.
    • Fix: add a render test asserting the incompatible-variable caution/error appears for a filter whose selected source kind its field can't use.
🔵 P3 nitpicks (1)
  • packages/app/src/components/AdhocDashboardFilter/AdhocDashboardFilter.tsx:163 -- when both suggestionsWhere.error and incompatibleVariables are present, the danger caution shows only suggestionsWhere.error, so the incompatible-variable reason is hidden.
    • Fix: combine both reasons in the tooltip, or document that the query error intentionally takes precedence.

Reviewers (2 returned + orchestrator analysis): previous-comments, learnings-researcher. The correctness, adversarial, kieran-typescript, testing, maintainability, api-contract, project-standards, and agent-native reviewers were dispatched but did not return results before synthesis was finalized; the findings above come from an independent full read of the diff plus the two completed reviewers. Treat correctness/adversarial coverage as partial.

Testing gaps:

  • No test for the isSourceOnOtherConnection / isSourceKindUnsupported tile placeholder states.
  • No component-level test for the incompatible/unresolved reference-variable caution and error rendering in the filter bar.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

Comment thread packages/app/src/DBDashboardPage.tsx Resolved
Comment thread packages/app/src/DBDashboardPage.tsx Resolved
Comment thread packages/app/src/DashboardFilters.tsx Resolved
Comment thread packages/app/src/DashboardFilters.tsx Resolved
Comment thread packages/app/src/DBDashboardPage.tsx Resolved
: []),
];
return (
<DashboardFilterSelect

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — CONNECTION filters fall through to an empty value dropdown, so a data source variable scoped to $conn never offers any source

Add a connection picker branch next to the isDataSourceFilter one (for example a ConnectionDashboardFilter built on ReferenceVariableSelect). As written, a CONNECTION filter skips the DATA_SOURCE and ADHOC branches and reaches DashboardFilterSelect. useDashboardFilterValues only fetches query-expression, static, and PromQL filters (hooks/useDashboardFilterValues.tsx:45-47), so that dropdown never gets options. The connection variable can then never be selected. Any DATA_SOURCE filter with connection: '$conn', which this PR resolves in useDashboardFilters, stays empty, and every tile that uses it stays blocked. The PR says these filters can be created through MCP, the API, or import. If the picker is planned for a later PR, hide or flag CONNECTION filters until then.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pulpdrew this seems worth addressing


// Reference variables are always single-select and required, so only the
// global requirement checkbox is consulted
if (isReferenceVariableFilterType(values.type)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The new CONNECTION save path can't be reached from the edit form

In DashboardFilterEditForm.tsx, add a formFilterType === 'CONNECTION' branch (it only needs the variable name input) and a CONNECTION entry in the type picker. Today a CONNECTION filter opened for editing falls to the default QueryExpressionFilterEditForm branch. That form's source and expression fields have rules={{ required: true }} (QueryExpressionFilterEditForm.tsx:134,187), so submitting is blocked before toSavedFilter runs. The type picker also shows a blank value because FILTER_TYPE_OPTIONS has no CONNECTION entry. The new CONNECTION branch in toSavedFilter and its unit test never run in the real UI.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

},
]
: []),
{ value: DashboardFilterType.enum.DATA_SOURCE, label: 'Data source' },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — 'Data source' filter type is offered unconditionally, but nothing in the UI can consume it yet

Put the DATA_SOURCE option behind a flag, the way IS_PROMQL_ENABLED and IS_ADHOC_FILTERS_ENABLED gate the options above it, until tiles and filters can reference it from the UI. The PR description says the variable can't be referenced from the UI yet. Users can still create one: an always-required picker that shows a 'required' warning in the filter bar. If they check 'Block every tile', it blocks every tile on the dashboard even though no tile can use it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

value: string | undefined,
sourceKindMap: ReadonlyMap<string, SourceKind>,
): React.ReactNode {
if (parseVariableReference(value) != null) return VARIABLE_ICON;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — getSourceOptionIcon's variable-reference branch can never run

Drop the parseVariableReference/VARIABLE_ICON branch, or add it in the PR that introduces a caller passing $name values. The only caller is ReferenceVariableDashboardFilter.tsx:150, and the values it passes are always concrete source IDs from the filter bar options or the validated selection. A reference like $logs can never reach it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

source.connection !== chart.config.connection;
// A variable's kinds can change after tiles use it, and saves don't check a
// source's kind against the tile's display type.
const isSourceKindUnsupported =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New tile resolution and placeholder logic has no tests

Add a component or E2E test where a tile's stored source is $var. It should cover: (a) the tile is blocked while the variable is unselected; (b) it queries the selected source once one is chosen; (c) tile-unsupported-source-kind shows for a source kind the chart type can't use; (d) the different-connection placeholder shows for raw SQL. Also check that onUpdateChart (for example a display-type switch) saves $var back, not the resolved ID. The new E2E spec only covers a fixed-source tile blocked by a global requirement. Nothing would catch restoreConfigReferences being dropped from onUpdateChart, which would silently overwrite the variable reference with a concrete source ID.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

This branch was successfully deployed

2 active deployments
Preview – hyperdx-storybook — 1017e511 Deployed Nov 18, 2026 by vercel[bot]
Preview – hyperdx-oss — 1017e511 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants