Skip to content

feat: support data source and connection variables in the external API and MCP - #3652

Merged
pulpdrew merged 5 commits into
drew/source-vars-1from
drew/source-vars-2
Nov 18, 2026
Merged

pulpdrew merged 5 commits into
drew/source-vars-1from
drew/source-vars-2

Conversation

@pulpdrew

@pulpdrew pulpdrew commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

PR 2/5 adding data source and data connection type variables. This PR adds external API and MCP support.

Source and connection variables allow for dashboards variable inputs that define a source or connection queried by a tile or other filters. Selected values are stored in source or connection id slots in the saved chart config, in the $varName format, then converted to source or connection IDs at runtime when there is a selection.

Variables are substituted prior to rendering the dashboard, at the top level, so that components and configs down the stack can just use source and connection IDs like they have been.

Direct $source references are not yet allowed in SQL tiles - the source is selected at the chart level and $__sourceTable can be used to reference it. Connection variables are mostly only useful for (a) raw sql tiles and (b) source variables that can be scoped to a single connection, dynamically.

How to test on Vercel preview

The E2E feature can be tested on PR 5: #3656

References

@pulpdrew
pulpdrew added this pull request to stack #3653 November 13, 2026 20:02
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 17, 2026 8:52pm UTC
hyperdx-storybook Ready Ready Preview Nov 17, 2026 8:52pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: ef79c91

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 464 passed • 1 skipped • 1121s

Status Count
✅ Passed 464
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

2 new clones introduced by this PR.

  • packages/api/src/mcp/tools/dashboards/queryTile.ts:69-87 ↔ packages/api/src/mcp/tools/dashboards/queryTiles.ts:175-193 — 19 lines
  • packages/api/src/utils/zod.ts:572-582 ↔ packages/api/src/utils/zod.ts:583-603 — 11 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 106 in total (1.15%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (2) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/dashboards.ts
    • packages/api/src/routers/external-api/v2/utils/dashboards.ts
  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 14
  • Production lines changed: 1223 (+ 1093 in test files, excluded from tier calculation)
  • Critical-path lines changed: 290
  • Branch: drew/source-vars-2
  • Author: pulpdrew

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

5 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 5 minor

3 posted as inline comment(s) on the changed lines. 2 listed below.

Findings outside the changed lines

2 minor
  • 🔵 packages/api/src/mcp/__tests__/dashboards/queryTile.int.test.ts:516 — No test checks that variableValues override the dashboard's saved selection → The schema says a DATA_SOURCE/CONNECTION value in variableValues "overrides the dashboard's saved selection" (resolveTileVariableReferences sets saved values first, then variableValues). But the variableValues int test creates a dashboard with no saved values, and every unit test in variables.test.ts passes [] for savedFilterValues. If the two loops were swapped, every test would still pass. Add a case with a saved selection (e.g. the trace source) plus a different variableValues selection, and assert which source wins: either the error text names the overriding ID or the returned rows come from that source. Also assert on the result content, not just isError being falsy.
  • 🔵 packages/api/src/routers/external-api/v2/dashboards.ts:2814 — OpenAPI docs omit $name support for PrometheusLabelFilterInput.sourceId and QueryExpressionFilterInput.appliesToSourceIds → This PR switches externalPromqlLabelFilterShape.sourceId and externalQueryExpressionFilterShape (whose appliesToSourceIds comes from the base schema) to accept variable references, and a test POSTs a PROMETHEUS_LABEL filter with sourceId: '$prom'. But the JSDoc here (and line 2626 for appliesToSourceIds), the generated openapi.json, and the MCP appliesToSourceIds description in packages/api/src/mcp/tools/dashboards/schemas.ts:1723 still describe these fields as plain source IDs. Add the same "or $name of a DATA_SOURCE filter" note that the other fields got.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This PR extends the external v2 API and MCP dashboard tools with DATA_SOURCE / CONNECTION variables, accepting $name references in sourceId / connectionId slots and resolving them to team-scoped source/connection IDs at query time. The resolution and validation paths are carefully written: references are stored verbatim, team scoping flows through getReferenceTargets(teamId), unresolved references are rejected before any ClickHouse query runs, and the selected source's kind/connection is validated against both the variable's offer and the tile's chart type. No data-loss, auth-bypass, injection, or guaranteed-crash path was found in the diff.

The substantive prior review comment (a PromQL tile silently running against a variable-selected log source) is confirmed addressed: getResolvedSourceError now converts the resolved tile to its internal config and rejects a selected source whose kind the tile cannot use.

🟡 P2 — recommended

  • packages/api/src/mcp/tools/dashboards/queryTiles.ts:235 — getReferenceTargets issues two team-wide reads (all sources + all connections) on every query_tiles/query_tile call even when no selected tile references a variable, and in the batch path it is awaited before deadlineAt is computed, so its latency sits outside the batch deadline budget.
    • Fix: Skip the fetch when no tile config in scope contains a $ reference, and compute the fetch latency within (or before accounting for) the batch deadline.
    • reliability, correctness
🔵 P3 nitpicks (2)
  • packages/api/src/mcp/tools/dashboards/variables.ts:242 — When a tile's sourceId references a name that resolves to a non-DATA_SOURCE variable (e.g. a text or connection variable), describeOffer returns the terse "no sources", producing a misleading error that does not explain the name is not a data source variable.

    • Fix: Distinguish "no such data source variable" from "no matching sources" in the error text.
  • packages/api/src/mcp/tools/dashboards/queryTiles.ts:175 — The duplicate-code checker flags the dashboard-fetch/convertToExternalDashboard boilerplate shared with queryTile.ts, and the near-identical externalDashboardPieChartConfigSchema / externalDashboardCategoricalBarChartConfigSchema in zod.ts; the latter is pre-existing structure (only the sourceId type changed here).

    • Fix: Optionally extract the shared tile-lookup helper; the zod clone is advisory and non-blocking.
    • maintainability, previous-comments

Reviewers: correctness, security, adversarial, api-contract, kieran-typescript, testing, maintainability, reliability, project-standards, previous-comments, agent-native, learnings-researcher.

Coverage note: At synthesis time, structured results had returned from the previous-comments and learnings reviewers; the remaining findings reflect the orchestrator's own diff analysis across the correctness, security, reliability, api-contract, maintainability, and typescript lenses. Independent verification found the OpenAPI schemas (DataSourceFilterInput, ConnectionFilterInput) consistent with the Zod shapes, the translation round-trip (translateFilterToExternalFilter / translateExternalFilterToFilter) symmetric, and reference-vs-ID comparisons in getSourceConnectionMismatches guarded correctly.

Testing gaps: Test coverage for the new paths is substantial (variables.test.ts, dashboards.int.test.ts, guards.test.ts). Consider adding an explicit unit test asserting getResolvedSourceError rejects a PromQL tile with a log-source variable selection, to lock in the behavior raised in prior review.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

.describe(
'Filter type discriminator for a variable whose value is a source.',
),
allowedSourceKinds: z

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Non-PromQL MCP filter schema lets a DATA_SOURCE variable offer promql sources

mcpDataSourceFilterSchema uses z.nativeEnum(SourceKind), and it is in both mcpDashboardFilterSchema and mcpPromqlDashboardFilterSchema. With PromQL off, an agent can still save allowedSourceKinds: ["promql"]. Every other filter shuts PromQL out of the non-PromQL schema (ADHOC sourceType is z.literal('sql')). isPromqlFilter in routers/external-api/v2/utils/dashboards.ts also doesn't hide these variables from hidePromql reads. Fix: use a non-PromQL variant of the schema (drop SourceKind.Promql) in the non-PromQL union.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* PROMETHEUS_LABEL: '#/components/schemas/PrometheusLabelFilterInput'
* SQL_QUERY: '#/components/schemas/SqlQueryFilterInput'
* ADHOC: '#/components/schemas/AdhocFilterInput'
* DATA_SOURCE: '#/components/schemas/DataSourceFilterInput'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — OpenAPI DATA_SOURCE/CONNECTION schemas omit fields that GET returns

translateFilterToExternalFilter returns these filters with minSelections: 1, maxSelections: 1, isBroadcastEnabled: false and isVariableEnabled: true. The strict external Zod shapes accept those fields, but DataSourceFilterInput and ConnectionFilterInput (reused by the Filter response schema) don't document them. PrometheusLabelFilterInput documents its fixed fields. Fix: add the four fields as single-value enums with defaults, so generated clients match what GET returns and PUT accepts.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@jordan-simonovski jordan-simonovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@pulpdrew
pulpdrew merged commit ef79c91 into main Nov 18, 2026
29 checks passed

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — ef79c916 Deployed Nov 17, 2026 by vercel[bot]
Preview – hyperdx-storybook — ef79c916 Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants