Skip to content

feat(api): classify failed ClickHouse proxy requests - #3705

Open
jordan-simonovski wants to merge 4 commits into
mainfrom
jordansimonovski/hdx-6072-classify-proxy-failures
Open

jordan-simonovski wants to merge 4 commits into
mainfrom
jordansimonovski/hdx-6072-classify-proxy-failures

Conversation

@jordan-simonovski

@jordan-simonovski jordan-simonovski commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Search & charts SLO can't tell an upstream hang from a customer's overloaded service or a bug in the proxy, and a query that fails mid-stream counts as a success. Every failed /clickhouse-proxy request now records why it failed, so the SLO can choose which failures count against it.

  • The server span gets hyperdx.clickhouse_proxy.error.category: upstream_timeout, upstream_unavailable, service_overloaded, query_timeout, query_error, client_aborted or proxy_error. The upstream status and the ClickHouse error code and type are recorded with it. For upstream errors that carry no ClickHouse code, such as an endpoint's 503 page, the first 1,000 characters of the body are recorded too, so a fast 503 from an idle service can be told apart from a full thread pool. ClickHouse's own messages are left out because they quote the query.
  • The clickhouse_proxy.query SLI carries the same value as error_category, so the SLO tile can filter on it.
  • A query that fails after ClickHouse has started streaming returns a 200 with the exception at the end of the body. It is now categorised like any other failure.
  • When the upstream breaks off mid-stream, the proxy error handler resets the connection instead of throwing ERR_HTTP_HEADERS_SENT, so the client can't read truncated rows as a complete result.

Breaking changes

  • None. outcome keeps its status-based meaning, and duration_ms still measures time to the upstream's headers.
  • One behaviour change: a 200 that the upstream cut off mid-stream used to be recorded twice (success at headers, then error). It is now recorded once, as an error.
Implementation detail
  • The outcome is recorded once per request, from close and errorMonitor listeners on the response. They are prepended because the HTTP instrumentation ends the server span from those same events.
  • Errors after a 200 are found by keeping the last 8 KB of the body. ClickHouse 25.11+ frames the exception with the X-ClickHouse-Exception-Tag value, which is parsed with @clickhouse/client-common. Older versions write it as an unmarked last line, matched only when that line starts with Code: N. DB::Exception, so quoted error text in row data doesn't match. Compressed bodies aren't inspected.
  • A request the browser abandons before ClickHouse answers gets the span attribute but no SLI event, matching what was recorded before.
  • upstream_timeout from the proxy's own timeout only fires when a proxy timeout is configured. None is set here today, so the category currently comes from ETIMEDOUT or an upstream 504.
  • Tests: a unit table for the classifier and the trailing-exception parser, and integration tests that drive the real proxy against a scripted upstream. Those cover a clean 200, untagged and tagged errors after a 200 (with the body passed through unchanged), the exception-code header, a 503 body, a cut stream, a refused connection and a client abort.

How to reproduce

Starting point: local integration tests (Docker).

On main:

  1. A 200 that ends in a ClickHouse exception is recorded as outcome: success with nothing on the span to say it failed.

On this branch:

  1. make dev-int FILE=clickhouseProxyOutcome
  2. Each scenario records the expected error_category on the clickhouse_proxy.query SLI.

How to test on Vercel preview

N/A, non-UI change.

References

Every failed /clickhouse-proxy request now records an error category on
its server span (upstream timeout, upstream unavailable, service
overloaded, query timeout, query error, client aborted, proxy error),
along with the upstream status, the ClickHouse error code and the error
message. The clickhouse_proxy.query SLI carries the category as
error_category, so the SLO can choose which failures count.

Errors ClickHouse writes at the end of a 200 response are detected too,
in both the tagged (25.11+) and untagged forms. Outcome and latency keep
their existing meaning.

The proxy error handler no longer writes a second status when the
upstream breaks off mid-stream, which threw ERR_HTTP_HEADERS_SENT.

HDX-6072
@jordan-simonovski
jordan-simonovski requested a review from a team as a code owner November 18, 2026 06:22
@changeset-bot

changeset-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c4c00bf

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 18, 2026 7:10am UTC
hyperdx-storybook Ready Ready Preview Nov 18, 2026 7:10am UTC

Request Review

@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 348 production lines changed (Tier 2 max: < 250)
  • Touches API routes or data models — hidden complexity risk

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 3
  • Production lines changed: 348 (+ 358 in test files, excluded from tier calculation)
  • Branch: jordansimonovski/hdx-6072-classify-proxy-failures
  • Author: jordan-simonovski

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 106 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 2 P3

No inline comments. 2 P3 folded at the bottom.

🔵 2 P3 — no action needed before merge
  • 🔵 P3 packages/api/src/utils/proxyOutcome.ts:58 — UNREACHABLE_ERRNOS duplicates the existing SERVER_NODE_ERROR_CODES set, and the two already differ → packages/api/src/mcp/tools/query/helpers.ts:838 (SERVER_NODE_ERROR_CODES) already lists the Node errnos that mean "upstream unreachable": ECONNREFUSED, ECONNRESET, ENOTFOUND, ETIMEDOUT, ENETUNREACH, EHOSTUNREACH, EPIPE and UND_ERR_CONNECT_TIMEOUT. The new set adds EAI_AGAIN and leaves out the timeout codes, so MCP and the proxy now classify the same failure differently. The ClickHouse timeout code is likewise already defined as CH_TIMEOUT_CODE in packages/api/src/tasks/checkAlerts/errors.ts:64. This breaks the REQUIRED convention "Before you add a type, schema, or helper". Fix: move one errno set into a shared module such as packages/api/src/utils/clickhouseErrors.ts, import it from both places, and keep ETIMEDOUT handled first, as classifyProxyFailure already does.
  • 🔵 P3 packages/api/src/utils/proxyOutcome.ts:853 — The proxy-timeout path never runs because no proxy timeout is configured → createProxyMiddleware in clickhouseProxy.ts sets neither proxyTimeout nor timeout, and the PR description says the same. So observeProxyRequest, the 'timeout' listener, ProxyState.timedOut, the timedOut && failedUpstream gate and the f.timedOut branch in classifyProxyFailure never change the result; upstream_timeout only ever comes from ETIMEDOUT or a 504. Fix: remove this plumbing until a proxy timeout is actually set, or configure proxyTimeout in the same change so the category is reachable and an integration test can cover it. Also correct the comment saying "Node emits 'timeout' on any idle socket": ClientRequest emits 'timeout' only when a socket timeout has been set.

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

ClickHouse quotes the query and its literals in exception messages, so
record the error type instead. Raw bodies are kept only for upstream
errors without a ClickHouse code, such as an endpoint's 503 page.
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 474 passed • 1 skipped • 1012s

Status Count
✅ Passed 474
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The change is well-structured and well-tested: outcome is recorded once (guarded by state.done), listeners are prepended so span attributes land before the HTTP instrumentation ends the server span, the error_category label is bounded/low-cardinality, and the res.destroy() path correctly replaces the previous ERR_HTTP_HEADERS_SENT throw. The findings below are accuracy and hardening recommendations, not merge blockers.

🟡 P2 — recommended

  • packages/api/src/utils/proxyOutcome.ts:236 — A client abort after the upstream has started streaming is classified as upstream_unavailable, not client_aborted: the downstream disconnect triggers the proxy aborted/error path, which sets upstreamCut/transportError, so failedUpstream is true and the clientAborted computation (!res.writableFinished && !failedUpstream) is false. Routine browser search re-runs that abort a streaming response then count against the upstream-availability signal the SLO is meant to isolate, and no test covers abort-after-headers (the hang scenario only exercises abort-before-headers).
    • Fix: Distinguish a downstream client disconnect from an upstream cut before classifying, and add an integration scenario that aborts after headers are received.
    • correctness, reliability, testing
🔵 P3 nitpicks (2)
  • packages/api/src/utils/proxyOutcome.ts:230 — findTrailingException (and the parseError/extractErrorAtTheEndOfChunk calls it makes) runs inside finishProxyOutcome, which executes in the response close/errorMonitor listeners with no surrounding try/catch, so any throw escapes as an uncaught exception on the response lifecycle.
    • Fix: Wrap the outcome computation in a try/catch so a parse or attribute error cannot escape the close listener.
  • packages/api/src/utils/proxyOutcome.ts:249 — The error.message span attribute records up to 1000 chars of the upstream error body when no ClickHouse code is found; a code-less multi-line error body whose exception line is not the last tail line can place query-quoting text on the span.
    • Fix: Scan the whole captured head/tail for a Code: N. DB::Exception marker before recording the body, and suppress the message when one is present.

Reviewers (11): correctness, reliability, security, api-contract, testing, maintainability, project-standards, adversarial, agent-native, learnings, previous-comments.

Testing gaps:

  • No coverage for a client abort that occurs after the upstream response headers have been received (only pre-header abort is tested).
  • proxy_error and upstream_timeout-via-ETIMEDOUT are covered only by the unit classifier table, not end-to-end.

Notes:

  • Prior PR comments are all automated bot output (changeset, Vercel, triage, jscpd, e2e); none are human review feedback requiring verification.
  • outcome intentionally stays status-based (a 200 with a trailing exception records outcome: success + error_category), which is documented behavior, not a defect.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

Drops an unsafe type assertion that pushed api lint past its warning
budget.
…error bodies

A 200 the upstream cuts off mid-stream is now an error on the SLI, the
idle-socket timeout only counts when the request failed, and the span
records the start of an upstream error page rather than its last 8 KB.

This branch was successfully deployed

2 active deployments
Preview – hyperdx-storybook — c4c00bf6 Deployed Nov 18, 2026 by vercel[bot]
Preview – hyperdx-oss — c4c00bf6 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant