Skip to content

feat: let tiles and filters use a data source variable as their source - #3655

Closed
pulpdrew wants to merge 4 commits into
drew/source-vars-3from
drew/source-vars-4
Closed

pulpdrew wants to merge 4 commits into
drew/source-vars-3from
drew/source-vars-4

Conversation

@pulpdrew

@pulpdrew pulpdrew commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

PR 4/5 adding data source and data connection type variables. This PR allows source variables to be referenced via the source dropdowns in the chart editor interface.

Source and connection variables allow for dashboards variable inputs that define a source or connection queried by a tile or other filters. Selected values are stored in source or connection id slots in the saved chart config, in the $varName format, then converted to source or connection IDs at runtime when there is a selection.

Variables are substituted prior to rendering the dashboard, at the top level, so that components and configs down the stack can just use source and connection IDs like they have been.

Direct $source references are not yet allowed in SQL tiles - the source is selected at the chart level and $__sourceTable can be used to reference it. Connection variables are mostly only useful for (a) raw sql tiles and (b) source variables that can be scoped to a single connection, dynamically.

Screenshots or video

Screen.Recording.2026-11-16.at.1.46.44.PM.mov

How to test on Vercel preview

The E2E feature can be tested on PR 5: #3656

References

@pulpdrew
pulpdrew added this pull request to stack #3653 November 13, 2026 20:04
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 18, 2026 12:51am UTC
hyperdx-storybook Ready Ready Preview Nov 18, 2026 12:51am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: a808f3c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Minor
@hyperdx/api Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 467 passed • 1 skipped • 1003s

Status Count
✅ Passed 467
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

4 new clones introduced by this PR.

  • packages/api/src/mcp/tools/dashboards/queryTile.ts:69-87 ↔ packages/api/src/mcp/tools/dashboards/queryTiles.ts:175-193 — 19 lines
  • packages/api/src/mcp/tools/trace/breakdown.ts:73-89 ↔ packages/api/src/mcp/tools/trace/waterfall.ts:339-354 — 17 lines
  • packages/api/src/utils/zod.ts:572-582 ↔ packages/api/src/utils/zod.ts:583-603 — 11 lines
  • packages/app/src/components/SourceMultiSelect.tsx:52-71 ↔ packages/app/src/components/SourceSelect.tsx:193-212 — 20 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 106 in total (1.14%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 850 production lines changed (Tier 2 max: < 250)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 17
  • Production lines changed: 850 (+ 846 in test files, excluded from tier calculation)
  • Branch: drew/source-vars-4
  • Author: pulpdrew

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 3 minor

3 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No P0/P1 issues confirmed by the reviewers that completed, and direct inspection of the two highest-risk paths — the alert drop-on-save and the source-picker grouping — found them correct. See the coverage caveat at the bottom: several bug-focused reviewers did not return in time, so this is not a clean bill of health.

The two prior-review blockers are addressed. The alert is no longer eagerly cleared from form state; chartFormResolver strips it only while the tile references a variable (EditTimeChartForm.tsx:185), react-hook-form hands the resolver's transformed values to the submit handler, and a warning banner (EditTimeChartForm.tsx:1516) tells the user — so switching back to a fixed source preserves the alert, and saving on a variable drops it as promised. Import of a tile referencing an undeclared $name is now blocked by getTemplateVariableReferenceIssues (DBDashboardImportPage.tsx:140).

🟡 P2 -- recommended

  • packages/app/src/components/DashboardFiltersModal/AdhocFilterEditForm.tsx:104 -- the new variable-reference branches (validateSourceKinds calling isSourceVariableReferenceAccepted, and tableConnections resolving refs then filtering by allowed kind) have no test; the existing form test never renders inside a ReferenceVariablesProvider.
    • Fix: Add a test rendering the form within a provider that selects a $var in the sources/appliesTo pickers and asserts kind-based acceptance plus resolved tableConnections.
    • testing
  • packages/app/src/DBDashboardImportPage.tsx:263 -- import carry-over of a variable reference inside a filter's appliesToSourceIds is untested; only the adhoc sources carry-over path is covered.
    • Fix: Add an import test with a template filter whose appliesToSourceIds includes a $var and assert the ref is kept unmapped while the plain name is mapped.
    • testing
🔵 P3 nitpicks (5)
  • packages/app/src/DBDashboardImportPage.tsx:245 -- new reference-issue error text uses raw c="red" instead of the mandated semantic variant="danger" for status text.
    • Fix: Render the error Text with variant="danger".
  • packages/app/src/components/ChartEditor/utils.ts:276 -- convertFormStateToSavedChartConfig prefers the stored ref (form.source || (source?.id ?? '')) while sibling convertFormStateToChartConfig prefers the resolved id; the asymmetry is intentional but easy to "unify" by mistake.
    • Fix: Extract named savedSourceRef/queriedSourceId helpers so the opposite precedence is self-documenting.
  • packages/app/src/components/SourceMultiSelect.tsx:52 -- the useFilteredSortedSourceItems + useWithSourceVariableItems + getSourceOptionIcon sequence duplicates SourceSelect.tsx:193 (flagged by jscpd).
    • Fix: Leave as-is or hoist the shared render-option wiring only if it stays readable; the shared logic is already centralized.
  • packages/app/src/components/DBEditTimeChartForm/EditTimeChartForm.tsx:408 -- the branch computing which missingSourceMessage variant to show (undeclared variable / no filter-bar selection / unusable kind) is not asserted at any level.
    • Fix: Add form tests for each of the three unsatisfied-reference states asserting the source field error text.
  • packages/app/src/components/DBEditTimeChartForm/useDebouncedFormValues.ts:12 -- new debounce-and-unsubscribe hook is only exercised indirectly through the form.
    • Fix: Add a fake-timer unit test asserting it emits getValues() after delay and unsubscribes on unmount.

Reviewers (6 of 11 returned): correctness-reviewer (did not return), testing-reviewer, maintainability-reviewer, project-standards-reviewer, agent-native-reviewer, learnings-researcher, kieran-typescript-reviewer (did not return), julik-frontend-races-reviewer (did not return), adversarial-reviewer (did not return), api-contract-reviewer (did not return), previous-comments-reviewer.

Coverage caveat: The correctness, adversarial, API/data-contract, TypeScript, and frontend-races reviewers had not returned when output was required, so logic-error, failure-scenario, persisted-config-contract, type-safety, and async-timing dimensions are only partially covered. The clean P0/P1 result reflects the completed reviewers plus direct inspection of the alert-save path (correct), the $name import-validation gate (addressed), and useWithSourceVariableItems grouping (useFilteredSortedSourceItems returns either a pure flat list or pure groups, never a mix, so the ...sourceItems spread preserves all groups) — it is not a substitute for the missing dimensions.

Testing gaps: Adhoc-filter variable-reference validation/resolution; import carry-over of refs in appliesToSourceIds; connection-variable (vs data-source) reference branch in getFormReferenceVariables; interchangeable-kind inference when a referenced variable has no selection; missingSourceMessage variant selection.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@jordan-simonovski jordan-simonovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment to fix, but otherwise looks good. Lmk when it's ready for re-review

}
}, [configType, displayType, previousDisplayType, setValue]);

useEffect(() => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This effect seems to remove the tile's alert as soon as a variable is picked, and nothing tells the user.

The alert has to go when the tile is saved on a variable, but clearing it in the form means picking $logs and switching back to the original source
still saves the tile without its alert. Dropping it at save time, plus a line of text where the alert section was, might be enough:

const savedConfig = convertFormStateToSavedChartConfig(form, tableSource, sourceKind);
if (savedConfig && usesReferenceVariable && 'alert' in savedConfig) {
  savedConfig.alert = undefined;
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, I've fixed in a808f3c. The same behavior impacts display type and config type switching, that could be handled separately as well

* A data source variable reference (`$name`) is portable as written, so
* import neither maps it nor asks the user to.
*/
const isVariableRef = (name: string | undefined) =>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens on import if a tile references $name and the template declares no data source variable with that name?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before it would have failed when saving due to API validations, now I have added front-end validations on the import page 👍

@jordan-simonovski jordan-simonovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

data-testid="source-selector"
allowedSourceKinds={allowedSourceKinds}
allowSourceVariables
connectionId={connection}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Raw SQL editor filters sources by the unresolved connection, while EditTimeChartForm resolves it

EditTimeChartForm now resolves the connection with useResolvedReference(connection) for allowedSourceItems and for the preview connection check. RawSqlChartEditor still passes the stored connection to SourceSelectControlled connectionId and to tableConnections (s.connection === connection). If a raw SQL tile's connection is a $conn reference, which getFormReferenceVariables treats as valid, the source picker lists only variables and SQL completions are empty. Fix: resolve it once (const resolvedConnection = useResolvedReference(connection)) and use that for connectionId, tableConnections and rawSqlConfig.connection.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

details: (
<Stack gap={0}>
{referenceIssues.map(issue => (
<Text key={issue} c="red">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New import error list uses raw c="red", Title Case copy, and keys that can collide

The conventions say new status text should use <Text variant="danger">, not c="red", and UI copy should be sentence case ("Failed to import dashboard"). Also, key={issue} collides when two tiles or filters have the same name and the same bad reference, because getTemplateVariableReferenceIssues builds identical strings for them. That causes React duplicate-key warnings. This block also repeats the schema-error block just above it. Fix: move both into one helper that renders string[] with variant="danger" and index-based keys.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* The required filters that block a preview with `applySelections` off: only
* the reference variables the tile can't be queried without.
*/
function getUnappliedPreviewBlockingFilterNames(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — getUnappliedPreviewBlockingFilterNames repeats the reference-variable matching in getBlockingRequiredFilters

packages/common-utils/src/dashboardFilterValues.ts:287-291 already pairs getReferenceVariableFilters(unsatisfiedRequiredFilters) with referenceVariableNames to find the blocking reference variables. This helper re-implements that matching in the app package. Fix: export the reference-variable subset from dashboardFilterValues (for example getBlockingReferenceVariableFilters) and call it here and inside getBlockingRequiredFilters, so the dashboard and the editor preview share one rule.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@pulpdrew
pulpdrew removed this pull request from stack #3653 November 18, 2026 01:06
@pulpdrew pulpdrew closed this Nov 18, 2026

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — a808f3c1 Deployed Nov 18, 2026 by vercel[bot]
Preview – hyperdx-storybook — a808f3c1 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants