Skip to content

feat: add connection variables for raw SQL tiles and connection-scoped data source variables - #3656

Closed
pulpdrew wants to merge 2 commits into
drew/source-vars-4from
drew/source-vars-5
Closed

pulpdrew wants to merge 2 commits into
drew/source-vars-4from
drew/source-vars-5

Conversation

@pulpdrew

@pulpdrew pulpdrew commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

PR 5/5 adding data source and data connection type variables. This PR adds a UI for defining, rendering, and referencing a connection variable in dashboard tiles (raw sql tiles) and filters (source variables).

Source and connection variables allow for dashboards variable inputs that define a source or connection queried by a tile or other filters. Selected values are stored in source or connection id slots in the saved chart config, in the $varName format, then converted to source or connection IDs at runtime when there is a selection.

Variables are substituted prior to rendering the dashboard, at the top level, so that components and configs down the stack can just use source and connection IDs like they have been.

Direct $source references are not yet allowed in SQL tiles - the source is selected at the chart level and $__sourceTable can be used to reference it. Connection variables are mostly only useful for (a) raw sql tiles and (b) source variables that can be scoped to a single connection, dynamically.

Screenshots or video

Screen.Recording.2026-11-16.at.3.26.32.PM.mov

How to test on Vercel preview

Create a dashboard, add some connection and source variables, try it out

References

@pulpdrew
pulpdrew added this pull request to stack #3653 November 13, 2026 20:04
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 18, 2026 12:58am UTC
hyperdx-storybook Ready Ready Preview Nov 18, 2026 12:58am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0a3f111

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/app Minor
@hyperdx/api Minor
@hyperdx/common-utils Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 469 passed • 1 skipped • 1016s

Status Count
✅ Passed 469
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

4 new clones introduced by this PR.

  • packages/api/src/mcp/tools/dashboards/queryTile.ts:69-87 ↔ packages/api/src/mcp/tools/dashboards/queryTiles.ts:175-193 — 19 lines
  • packages/api/src/mcp/tools/trace/breakdown.ts:73-89 ↔ packages/api/src/mcp/tools/trace/waterfall.ts:339-354 — 17 lines
  • packages/api/src/utils/zod.ts:572-582 ↔ packages/api/src/utils/zod.ts:583-603 — 11 lines
  • packages/app/src/components/SourceMultiSelect.tsx:52-71 ↔ packages/app/src/components/SourceSelect.tsx:199-218 — 20 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 106 in total (1.14%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (1) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/dashboards.ts
  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 17
  • Production lines changed: 407 (+ 432 in test files, excluded from tier calculation)
  • Critical-path lines changed: 8
  • Branch: drew/source-vars-5
  • Author: pulpdrew

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

2 minor
  • 🔵 packages/common-utils/src/types.ts:2449 — Schema doc comments still say a SQL query filter's connection is an ID, and connection variables are used only by tiles and data source variables → Update the connection comment on SqlQueryDashboardFilterSchema to say it is a connection ID or a $name reference to a CONNECTION filter. Also add SQL query filters to the list of referrers in the ConnectionDashboardFilterSchema comment (~line 2515), to match the OpenAPI, MCP and getFilterReferenceFields changes in this PR.
  • 🔵 packages/app/src/__tests__/DBDashboardImportPage.test.tsx:672 — Import test never covers a SQL query filter whose connection is $conn, the case the head commit adds → Add a SQL_QUERY filter with connection: '$conn' to connectionVariableTemplate. Assert it gets no connection-mapping row, that finishing the import doesn't require one, and that payload.filters[n].connection stays '$conn'. Today that filter goes through the generic source-filter branch at DBDashboardImportPage.tsx:1241, and no test covers that path.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

The core risk in this PR — widening connectionId for SQL_QUERY filters to accept a $variable reference while validateFilterConnections skips references — is correctly compensated. Reference connections are validated by getVariableReferenceError → validateDashboardVariableReferences, which is wired into both the internal save path (packages/api/src/routers/api/dashboards.ts:194,243) and the external v2 path (packages/api/src/routers/external-api/v2/utils/dashboards.ts:1823, after convertExternalFiltersToInternal translates connectionId→connection). Non-reference connection IDs still flow through the per-team getConnectionsByTeam ownership check, so tenancy is preserved. The common-utils validation layer has direct tests for accepting a declared connection variable and rejecting an undeclared one, including the SQL_QUERY connection path.

🟡 P2 — recommended

  • packages/api/src/routers/api/dashboards.ts:111 — validateFilterConnections now skips variable references for team-ownership checking; the compensating ownership guarantee lives entirely in validateDashboardVariableReferences, so the two must stay in lockstep.
    • Fix: Add an API integration test asserting a SQL_QUERY filter with a non-team connection ID is still rejected and one with a $conn reference to a declared CONNECTION filter is accepted, so a future refactor of the reference-validation path cannot silently reopen the cross-team gap.
    • correctness, security
🔵 P3 nitpicks (3)
  • packages/app/src/DBDashboardImportPage.tsx:1237 — findConnection(data.filterConnectionMappings?.[idx])!.id (and the tile variant at :1186) relies on the form superRefine guaranteeing a resolvable mapping; if the selected mapping fails to resolve against the loaded connections, the non-null assertion throws at submit rather than surfacing a validation message.
    • Fix: Resolve the connection into a local before use and surface a user-facing error when it is absent instead of asserting non-null.
  • packages/app/src/components/sourceSelectUtils.tsx:218 — the connection/variable items wiring duplicates the source-variable items shape (jscpd also flags SourceMultiSelect.tsx ↔ SourceSelect.tsx), and this file has grown to ~322 lines, past the project's 300-line guideline.
    • Fix: Extract the shared "Variables" group builder into one helper and consider splitting the pickers' item-assembly utilities out of sourceSelectUtils.tsx.
  • packages/api/src/utils/zod.ts:583 — the objectIdOrVariableReferenceSchema usage repeats the inline object-ID-or-reference shape that jscpd flags as a new clone at zod.ts:572-603.
    • Fix: Reuse the single shared schema rather than redefining the union per field.

Reviewers (10 dispatched): correctness, adversarial, security, api-contract, kieran-typescript, frontend-races, testing, maintainability, previous-comments, project-standards. Previous-comments and project-standards returned clean; previous-comments confirmed the sourceSelectUtils.tsx:133 connection-unlimited-variable trade-off is consciously deferred (per the author's reply on the thread), not an unaddressed regression. Remaining findings reflect the orchestrator's own verification of the validation wiring, tenancy guards, and test coverage.

Testing gaps: API-level test that the validateFilterConnections reference-skip still enforces team ownership for literal connection IDs on SQL_QUERY filters; confirm the import/mapping page covers the new connection-scoped DATA_SOURCE variable path (not only SQL_QUERY).

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@pulpdrew
pulpdrew force-pushed the drew/source-vars-4 branch 2 times, most recently from 6e0d854 to a3c9c2f Compare November 17, 2026 17:04
@github-actions github-actions Bot added review/tier-4 Critical — deep review + domain expert sign-off and removed review/tier-3 Standard — full human review required labels Nov 17, 2026
const renderConnectionIcon = () => <IconServer size={16} />;

/** A connection variable's picker. */
export const ConnectionDashboardFilter = ({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — A connection variable pointing to a deleted connection looks unselected, but tiles keep querying it

ReferenceVariableSelect hides a value that is no longer in the list (lines 67-71), so the new ConnectionDashboardFilter shows the 'required' warning when the saved connection ID has been deleted. But useDashboardFilters.tsx:252-257 builds connectionValues and referenceVariableValues without checking the ID against existing connections. It does check data source variables against the existing sources (lines 258-277). As a result, tiles, SQL query filters and data source variables limited to $conn keep resolving to the deleted ID instead of being blocked as missing a selection. Fix: drop a connection variable selection whose ID is not in useConnections() data, the same way data source selections are dropped.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

isSourceVariableAccepted(variable, allowedSourceKinds),
(!allowedSourceKinds ||
isSourceVariableAccepted(variable, allowedSourceKinds)) &&
(!connection ||

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should a data source variable with no connection limit be offered to a raw SQL tile that runs on a fixed connection?

It passes this filter, so a viewer can pick a source on another connection and the tile shows the "different connection" placeholder. Is that the intended trade-off, or would it be better to offer only variables limited to the tile's connection?

(!connection || variable.connection === connection)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the intended trade-off for now.

Something I want to add as a follow up is the ability for the connection input to refer to a source variable, basically meaning "use the connection associated with this source." I want to offer the flexibility of changing both connection and source, but it's currently a little annoying to have to change both. Once that is in place, I would hope that there are few cases of users with a fixed connection and a variable source that may not belong to that connection.

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 0a3f111c Deployed Nov 18, 2026 by vercel[bot]
Preview – hyperdx-storybook — 0a3f111c Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants