Repository navigation
feat(api): classify failed ClickHouse proxy requests - #3705
jordan-simonovski wants to merge 4 commits into
Conversation
Every failed /clickhouse-proxy request now records an error category on its server span (upstream timeout, upstream unavailable, service overloaded, query timeout, query error, client aborted, proxy error), along with the upstream status, the ClickHouse error code and the error message. The clickhouse_proxy.query SLI carries the category as error_category, so the SLO can choose which failures count. Errors ClickHouse writes at the end of a 200 response are detected too, in both the tagged (25.11+) and untagged forms. Outcome and latency keep their existing meaning. The proxy error handler no longer writes a second status when the upstream breaks off mid-stream, which threw ERR_HTTP_HEADERS_SENT. HDX-6072
🦋 Changeset detectedLatest commit: c4c00bf The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🟡 Tier 3 — StandardIntroduces new logic, modifies core functionality, or touches areas with non-trivial risk. Why this tier:
Review process: Full human review — logic, architecture, edge cases. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
PR Review
2 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 2 P3 No inline comments. 2 P3 folded at the bottom. 🔵 2 P3 — no action needed before merge
Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides. |
ClickHouse quotes the query and its literals in exception messages, so record the error type instead. Raw bodies are kept only for upstream errors without a ClickHouse code, such as an endpoint's 503 page.
E2E Test Results✅ All tests passed • 474 passed • 1 skipped • 1012s
Tests ran across 4 shards in parallel. |
Deep Review✅ No critical issues found. The change is well-structured and well-tested: outcome is recorded once (guarded by 🟡 P2 — recommended
🔵 P3 nitpicks (2)
Reviewers (11): correctness, reliability, security, api-contract, testing, maintainability, project-standards, adversarial, agent-native, learnings, previous-comments. Testing gaps:
Notes:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
Drops an unsafe type assertion that pushed api lint past its warning budget.
…error bodies A 200 the upstream cuts off mid-stream is now an error on the SLI, the idle-socket timeout only counts when the request failed, and the span records the start of an upstream error page rather than its last 8 KB.
Summary
The Search & charts SLO can't tell an upstream hang from a customer's overloaded service or a bug in the proxy, and a query that fails mid-stream counts as a success. Every failed
/clickhouse-proxyrequest now records why it failed, so the SLO can choose which failures count against it.hyperdx.clickhouse_proxy.error.category:upstream_timeout,upstream_unavailable,service_overloaded,query_timeout,query_error,client_abortedorproxy_error. The upstream status and the ClickHouse error code and type are recorded with it. For upstream errors that carry no ClickHouse code, such as an endpoint's 503 page, the first 1,000 characters of the body are recorded too, so a fast 503 from an idle service can be told apart from a full thread pool. ClickHouse's own messages are left out because they quote the query.clickhouse_proxy.querySLI carries the same value aserror_category, so the SLO tile can filter on it.ERR_HTTP_HEADERS_SENT, so the client can't read truncated rows as a complete result.Breaking changes
outcomekeeps its status-based meaning, andduration_msstill measures time to the upstream's headers.Implementation detail
closeanderrorMonitorlisteners on the response. They are prepended because the HTTP instrumentation ends the server span from those same events.X-ClickHouse-Exception-Tagvalue, which is parsed with@clickhouse/client-common. Older versions write it as an unmarked last line, matched only when that line starts withCode: N. DB::Exception, so quoted error text in row data doesn't match. Compressed bodies aren't inspected.upstream_timeoutfrom the proxy's own timeout only fires when a proxy timeout is configured. None is set here today, so the category currently comes fromETIMEDOUTor an upstream 504.How to reproduce
Starting point: local integration tests (Docker).
On
main:outcome: successwith nothing on the span to say it failed.On this branch:
make dev-int FILE=clickhouseProxyOutcomeerror_categoryon theclickhouse_proxy.querySLI.How to test on Vercel preview
N/A, non-UI change.
References