Skip to content

feat(app): show RED metrics on the trace search results view - #3183

Open
alex-fedotyev wants to merge 10 commits into
mainfrom
alex/red-metrics-trace-search-v2
Open

alex-fedotyev wants to merge 10 commits into
mainfrom
alex/red-metrics-trace-search-v2

Conversation

@alex-fedotyev

@alex-fedotyev alex-fedotyev commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Stack

Layer 1 of 2 in a GitHub stack (stack #3688). GitHub shows each layer's base as the branch below it, but the stack's base is main: merging any PR lands it and every PR below it on main, one squashed commit each. Each PR's diff is only its own layer, and CI runs as if each targets main:

  1. feat(app): show RED metrics on the trace search results view #3183 RED metrics on the trace search results view (this PR)
  2. feat(app): add a duration heatmap view to trace search #3184 Duration heatmap view for trace search

Summary

Trace search shows a single count histogram above the results. For trace sources that expose a duration column, this replaces it with RED metrics:

  • Throughput is the status-grouped span count (bars). It renders the same count the histogram did and reuses that query, so the results-count header and this tile share a single scan, and clicking a status series filters the results.
  • Errors toggles between rate (countIf(error) / count(), rendered as a percent line) and volume (countIf(error), bars, grouped by status).
  • Duration shows Avg, P95, and P99 over the source's raw duration column (line), with the unit applied at display from the source's durationPrecision.

Every aggregation is over a raw column (count, countIf, quantile/avg of the duration expression), so AggregatingMergeTree materialized views can satisfy the queries; the ratio and the unit conversion happen at the display layer or in a post-aggregation column.

The three charts are DBTimeCharts under a shared sync scope, so hovering one shows a synced cursor on all three at the same timestamp, and each renders as a dashboard-tile card. Clicking a status series on the Throughput or Errors-volume chart filters the results, which is the drill-down the histogram offered. Logs and other sources keep the existing histogram, as do trace sources on a compact (narrow/mobile) viewport, where three side-by-side tiles would not fit.

The per-chart aggregations for Errors and Duration live in a small pure module, traceRedMetrics.ts, built from the same base config the histogram uses, so they honor the active WHERE filter and time range and can be unit tested without rendering.

Three supporting props on shared chart components, each with its only consumer in this PR:

  • yAxisMaxDomain, so the Errors rate tile caps at 100% instead of auto-scaling a near-zero series to a nonsense range. The cap is a numeric domain, so the axis keeps clean, de-duplicated tick labels and never collapses to a single point.
  • compactXAxisLabels, so the edge time labels do not clip on tiles this narrow. The labels keep the axis color the y-axis labels use.
  • applySelectColors, so the Duration tile can pin its percentile colors (below) without changing how any other chart colors its series.

The Duration percentiles are pinned to cool hues (Avg blue, P95 cyan, P99 purple) so latency never borrows the red the Errors line uses, and each percentile keeps its color across sessions regardless of how the data sorts the lines. The colors are set on each series' select-item color (the schema's existing per-series color field), so there is a single source of truth and no separate color map to keep in sync. DBTimeChart only reads them when the caller passes applySelectColors: saved tiles only set color through the table editor, so their line and bar charts keep the positional palette, and grouped series never collapse to one color. Tokens resolve the way the table and number tiles resolve them, so a legacy token maps to its hue and an unknown one falls back to the palette instead of failing the chart.

The Errors and Duration tiles are auxiliary to the results query, so they run under their own query namespace: that keeps the slow quantile scan out of the reported search latency, while live tail still waits on them before advancing the window, so a tick cannot stack a second set of chart queries on the first. Throughput shares the results-count query instead, so it adds no extra scan.

Screenshots

Light theme:

tiles light

Dark theme:

tiles dark

Test plan

  • Unit tests for the aggregation builders (traceRedMetrics.test.ts) and the chart color pipeline (ChartUtils.test.ts), including that a select-item color overrides the positional palette and does not consume a positional slot, and that getSelectSeriesColors keys colors by alias, maps a legacy token to its hue, and skips unknown tokens and unaliased items.
  • A Duration test pins each percentile to a stable, non-reserved hue (Avg/P95/P99 to blue/cyan/purple), so latency never falls back to the reserved chart-red slot. Because the color now lives on the select item, it cannot drift from the series it colors.
  • computeYAxisBounds coverage for the capped rate axis: a flat/zero series shows [0, cap] with clean ticks, the cap holds under selection (no [0, 0] collapse) and over reference lines, and is skipped for stacked charts.
  • Full app unit suite green, plus tsc --noEmit, eslint with no new warnings, and knip. CI covers the full suite and the production build:clickhouse bundle.
  • Verified on the stack's Vercel preview against the demo traces dataset, in light and dark: the three tiles render with data, the Duration lines are blue, cyan and purple with no red, the x- and y-axis labels share one color, Rate/Volume switches, and a Logs source keeps the single histogram. With a WHERE that matches nothing the tiles render empty axes rather than a nonsense scale; when the search query fails the chart area hides and the query error shows, as with the histogram; and a compact (390px) viewport or a custom group-by falls back to the histogram.

Notes

@vercel

vercel Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 18, 2026 5:57am UTC
hyperdx-storybook Ready Ready Preview Nov 18, 2026 5:57am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 78a275e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Minor
@hyperdx/api Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@greptile-apps

greptile-apps Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no actionable new issue was found in the changes since the previous review.

Summary

The PR replaces the trace-search count histogram with synchronized Throughput, Errors, and Duration charts while retaining the existing histogram for other sources.

  • Builds RED aggregations from the active search configuration and source expressions.
  • Adds stable per-series colors, compact time-axis labels, and capped percentage axes to shared chart components.
  • Separates auxiliary chart queries from search-latency telemetry while coordinating them with live-tail refresh.
  • Centralizes the trace status error predicate and distinguishes it from log-severity handling.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Search[Trace search state] --> Base[Shared filtered chart config]
    Base --> Throughput[Throughput: count spans]
    Base --> Errors[Errors: rate or volume]
    Base --> Duration[Duration: avg, p95, p99]
    Throughput --> Sync[Shared hover synchronization]
    Errors --> Sync
    Duration --> Sync
    Errors -->|Volume series click| Filter[Apply status filter to results]
    REDQueries[RED query namespace] --> LiveTail[Pause live-tail window advancement]
Loading

Reviews (4) · Last reviewed commit: "refactor(app): follow the error predicat..."

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The RED-metrics view is built from well-factored pure builders (traceRedMetrics.ts) with solid unit coverage, and the risky mechanics verified clean on close reading: the error-rate column divides error_spans / total_spans only over returned buckets (always >= 1, gap-fill is client-side), getSelectSeriesColors keys by select-item alias and setLineColors looks it up by the matching result-column name, unknown palette tokens are filtered through resolveChartPaletteToken so they fall back to the positional palette instead of throwing, and the RED_QUERY_KEY_PREFIX is a distinct first key element so it stays out of search-prefixed telemetry while isRedQueryFetching still gates live-tail. The recommendations below are test-coverage gaps on the new wiring, not defects in the diff.

🟡 P2 — recommended

  • packages/app/src/components/Search/TraceRedMetricsChart.tsx:1 — The component's new user-facing behavior is untested; only the pure config builders have unit coverage, so the Rate/Volume toggle, legend gating via isUsingGroupBy, per-mode onFocusSeries/yAxisMaxDomain/hiddenSeries, and the key={errorsMode} remount could regress silently.
    • Fix: Add a render test asserting the three tiles mount, the Rate/Volume switch swaps display type and axis cap, and onFocusSeries is wired only on the status-grouped tiles.
    • testing, correctness
  • packages/app/src/DBSearchPage.tsx:2954 — The RED-vs-histogram selection (traceRedMetricsSource != null && !isCompact && !isCustomGroupBy) is untested, so a regression in any of the three conditions would silently render the wrong chart for logs, compact viewports, or custom group-by.
    • Fix: Add coverage (or extract the predicate into a testable helper) asserting each condition falls back to the single histogram.
    • testing
🔵 P3 nitpicks (1)
  • packages/app/src/DBSearchPage.tsx:1757 — The live-tail pause coordination added via isRedQueryFetching (so ticks don't stack quantile scans) has no regression test, leaving the pause/no-op-off-trace-view behavior unguarded.
    • Fix: Add a test asserting pause goes true while a RED-prefixed query is in flight and stays false when the tiles are unmounted.

Reviewers (12): correctness, testing, maintainability, project-standards, agent-native, learnings-researcher, performance, reliability, adversarial, previous-comments, kieran-typescript, frontend-races.

Testing gaps:

  • No component/render test for TraceRedMetricsChart (mode toggle, legend, drill-down wiring, remount-on-key).
  • No test for the DBSearchPage RED-vs-histogram fallback branching.
  • No regression test for the isRedQueryFetching live-tail coordination.
  • Throughput's shared-scan dedup with the results-count query depends on an exact query-key/config match; a divergence would silently double-scan rather than fail — not currently asserted by a test.

Note: ce-learnings-researcher found no applicable prior solutions in the repo's learnings store; durable decisions here (palette-token pinning, capped y-axis domain rules, query-key-prefix dedup, live-tail coordination) are good /ce-compound candidates.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

if (mode === 'rate') {
return {
...base,
select: [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Error-rate count + countIf + post-aggregation ratio + hidden helper series is a fourth copy of an existing pattern

The repo's REQUIRED DRY rule asks for a grep on the operation before adding a helper. This exact trio already exists at ServicesDashboardPage/HttpTab.tsx:221-239 (requestErrorRateConfig: total_requests / error_requests / error_rate, ERROR_RATE_PERCENTAGE_NUMBER_FORMAT) with its matching hiddenSeries={['total_requests','error_requests']} at HttpTab.tsx:419, again inside the CTE at HttpTab.tsx:282-298, and again at llm/dashboard/OverviewCharts.tsx:152-166. Extract one builder — e.g. errorRateSelect(errorCondition, { totalAlias, errorAlias, rateAlias }) returning the select trio plus its hidden-series list — into a shared module and have all four call sites use it, rather than adding a fourth independently-aliased copy whose helper names must be kept in sync with a separate ERROR_RATE_HELPER_SERIES constant by hand.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shape is shared, agreed, but extracting a builder would touch HttpTab, OverviewCharts and the service-dashboard side panels, which widens this PR beyond its own files. Tracking it as a follow-up rather than folding it in here.

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 1 P3

No inline comments. 1 P3 folded at the bottom.

🔵 1 P3 — no action needed before merge
  • 🔵 P3 packages/app/src/components/Search/__tests__/traceRedMetrics.test.ts:1321 — The 'non-reserved hue' test adds no coverage → Drop this test or make it check something new. The toEqual in the test just above already pins the exact color of every select item, and the reserved-hue filter can only pass once the colors match that exact list. It does not test the actual risk, which is the rendered lineData color through DBTimeChart with applySelectColors.

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@alex-fedotyev
alex-fedotyev force-pushed the alex/red-metrics-trace-search-v2 branch from 7c3a45a to 108e06a Compare September 24, 2026 02:34
@alex-fedotyev
alex-fedotyev changed the base branch from alex/duration-heatmap-format to alex/error-span-predicate September 24, 2026 02:34
@alex-fedotyev
alex-fedotyev changed the base branch from alex/error-span-predicate to main September 24, 2026 02:34
Comment thread packages/app/src/ChartUtils.tsx Resolved
@alex-fedotyev alex-fedotyev reopened this Sep 24, 2026
@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 700 production lines changed (Tier 2 max: < 250)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 6
  • Production lines changed: 700 (+ 491 in test files, excluded from tier calculation)
  • Branch: alex/red-metrics-trace-search-v2
  • Author: alex-fedotyev

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 473 passed • 1 skipped • 1452s

Status Count
✅ Passed 473
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 106 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

SpencerTorres
SpencerTorres previously approved these changes Nov 18, 2026
alex-fedotyev and others added 10 commits November 17, 2026 21:54
Trace search showed a single count histogram above the results. For trace
sources with a duration column it now shows Throughput, Errors and Duration
side by side, under a shared sync scope so hovering one moves the cursor on
all three.

Each chart is built from the same base config the histogram uses, in a pure
module so the aggregations can be unit tested without rendering. Every
aggregation is over a raw column so materialized views can satisfy it; the
error ratio and the duration unit are applied at display instead.

Logs and other sources keep the existing histogram.
- Fall back to the single histogram on a compact (mobile) viewport
  instead of three unreadable side-by-side tiles.
- Compute a numeric capped y-axis domain for the rate tile so it dedupes
  tick labels and no longer collapses to [0, 0] when the sole series is
  selected; the cap now applies on the selection and fit paths too.
- Drop the unreachable 0/0 guard on the error-rate column; a returned
  GROUP BY bucket always has total_spans >= 1, so divide directly the
  way HttpTab does.
- Rename the Duration percentile series p95/p99 to P95/P99 to match the
  app's acronym casing.
- Strengthen the volume-mode test so it catches dropped sidebar filters,
  and add computeYAxisBounds coverage for the capped axis.
- Throughput now renders the status-grouped count histogram this view
  replaces, under the results-count query-key prefix, so react-query runs
  a single shared scan instead of a second ungrouped count() on every
  search and live-tail tick. This also restores the per-status drill-down
  the histogram offered.
- Duration series pin their colors through each select item's `color`
  rather than a name-keyed seriesColors prop, so there is one source of
  truth for per-series color. Drops the DBTimeChart seriesColors prop, the
  DURATION_SERIES_COLORS map, and the alias/color lockstep test;
  setLineColors reads the color off the select item by result-column name.
…ant duration format

- When a trace source has a custom group-by selected, fall back to the
  single histogram (which honors arbitrary group-by) instead of the RED
  tiles, whose grouping is fixed (Throughput and Errors by status,
  Duration ungrouped). The group-by selector now has a coherent effect
  rather than silently regrouping only part of the view.
- Drop the hand-computed duration numberFormat: DBTimeChart already infers
  it via useChartNumberFormats -> getTraceDurationNumberFormat, because
  each duration series aggregates the source's duration expression. Removes
  the getTraceDurationNumberFormat memo and durationConfig's numberFormat
  parameter.
- Compact x-axis ticks use the responsive axis tick font size instead of a
  hardcoded 11px, matching the non-compact ticks and honoring the user's
  content font scale.
- Correct the yAxisMaxDomain prop doc: the cap applies on every path
  (selection and fit included), not only the default path.
…ries

Group-by none resolves to no grouping, which collapses Throughput to a
single count() series; with the legend on, that surfaced the raw aggregate
expression ("count()") as the series label. Show the Throughput legend only
when the view is grouped (the status-code default), where it names each
status, matching how the histogram gated its own legend. The ungrouped case
keeps just the "Throughput" title.
Reading select[].color in every DBTimeChart changed saved dashboard
tiles. The field is only set through the table editor, and other display
types ignore it, so a tile switched from Table to Line took on its column
colors, and every group of a grouped series shared one color.

- DBTimeChart applies select-item colors only with applySelectColors,
  which the trace RED duration tile sets. Every other chart keeps the
  positional palette.
- Derive the colors in getSelectSeriesColors, which resolves each token
  the way the table and number tiles do: a legacy token maps to its hue,
  and an unknown one falls back to the palette instead of failing the
  chart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The compact tick renderer hardcoded var(--color-text-muted), while the
y-axis and regular x-axis ticks use the fill Recharts passes in, so the
RED tiles showed their x and y labels in different colors. Use the
passed-in fill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The grouped check re-implemented isUsingGroupBy from common-utils. Its
extra whitespace branch could never run, since the search group-by is
already trimmed before it reaches the chart config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test relies on seriesHi taking the first palette slot and said that
came from its higher peak. Colors follow the sorted series order, which
falls back to the series key when log-level order ties, so that is why
seriesHi comes first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 78a275eb Deployed Nov 18, 2026 by vercel[bot]
Preview – hyperdx-storybook — 78a275eb Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants