From 0fd2593a8d8cb5deb604d0d61108852b547e938e Mon Sep 17 00:00:00 2001 From: Jordan Simonovski Date: Wed, 18 Nov 2026 17:21:06 +1100 Subject: [PATCH 1/4] feat(api): classify failed ClickHouse proxy requests Every failed /clickhouse-proxy request now records an error category on its server span (upstream timeout, upstream unavailable, service overloaded, query timeout, query error, client aborted, proxy error), along with the upstream status, the ClickHouse error code and the error message. The clickhouse_proxy.query SLI carries the category as error_category, so the SLO can choose which failures count. Errors ClickHouse writes at the end of a 200 response are detected too, in both the tagged (25.11+) and untagged forms. Outcome and latency keep their existing meaning. The proxy error handler no longer writes a second status when the upstream breaks off mid-stream, which threw ERR_HTTP_HEADERS_SENT. HDX-6072 --- .../classify-clickhouse-proxy-failures.md | 8 + agent_docs/observability.md | 72 ++--- .../clickhouseProxyOutcome.int.test.ts | 239 ++++++++++++++++ .../api/src/routers/api/clickhouseProxy.ts | 57 ++-- .../src/utils/__tests__/proxyOutcome.test.ts | 107 +++++++ packages/api/src/utils/instrumentation.ts | 12 +- packages/api/src/utils/proxyOutcome.ts | 270 ++++++++++++++++++ 7 files changed, 692 insertions(+), 73 deletions(-) create mode 100644 .changeset/classify-clickhouse-proxy-failures.md create mode 100644 packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts create mode 100644 packages/api/src/utils/__tests__/proxyOutcome.test.ts create mode 100644 packages/api/src/utils/proxyOutcome.ts diff --git a/.changeset/classify-clickhouse-proxy-failures.md b/.changeset/classify-clickhouse-proxy-failures.md new file mode 100644 index 000000000..5fbfc2b24 --- /dev/null +++ b/.changeset/classify-clickhouse-proxy-failures.md @@ -0,0 +1,8 @@ +--- +'@hyperdx/api': patch +--- + +feat: Classify failed ClickHouse proxy requests on the server span and the +`clickhouse_proxy.query` SLI (`error_category`), including errors ClickHouse +writes after a 200 response. The span also carries the upstream status, the +ClickHouse error code and the error message. diff --git a/agent_docs/observability.md b/agent_docs/observability.md index 1de5356b4..b8950ac54 100644 --- a/agent_docs/observability.md +++ b/agent_docs/observability.md @@ -1,8 +1,8 @@ # Observability & Instrumentation -ClickStack is an observability product, so our own code should be a shining example -of well-instrumented software. When you add or change a feature, instrument it -as part of the change - not as an afterthought. +ClickStack is an observability product, so our own code should be a shining +example of well-instrumented software. When you add or change a feature, +instrument it as part of the change - not as an afterthought. This guide covers the conventions and the shared helpers. The principles are aligned with the @@ -11,9 +11,9 @@ aligned with the ## TL;DR (the non-negotiables) 1. **Every team-scoped operation carries team + user context.** Use - `setBusinessContext(...)` so `hyperdx.team.id` / `user.id` end up on - the trace. Auth middleware already does this for HTTP requests; background - jobs and other entry points must do it themselves. + `setBusinessContext(...)` so `hyperdx.team.id` / `user.id` end up on the + trace. Auth middleware already does this for HTTP requests; background jobs + and other entry points must do it themselves. 2. **If something is worth a log, put it on the span first.** The active span is the wide event for the current unit of work — attach the fact there as an attribute. If it also marks a countable event worth aggregating (an error, a @@ -30,14 +30,15 @@ aligned with the already in flight, and put point-in-time / gauge-style values (sizes, counts, depths, current state) there rather than in a gauge. This is an internal engineering preference, not a rule against metrics — counters and histograms - remain first-class, feed alerts and SLOs, and many ClickStack deployments depend - heavily on them. See [Wide events over gauges](#wide-events-over-gauges). + remain first-class, feed alerts and SLOs, and many ClickStack deployments + depend heavily on them. See + [Wide events over gauges](#wide-events-over-gauges). 5. **Use the shared helpers** in [`packages/api/src/utils/instrumentation.ts`](../packages/api/src/utils/instrumentation.ts). Don't hand-roll tracer/meter lifecycle. -6. **A new surface that queries ClickHouse declares its query attribution.** - One provider or one `withQueryAttribution` scope at the entry point, not a - change at each query. See [Query attribution](#query-attribution). +6. **A new surface that queries ClickHouse declares its query attribution.** One + provider or one `withQueryAttribution` scope at the entry point, not a change + at each query. See [Query attribution](#query-attribution). ## The shared helper library @@ -116,7 +117,7 @@ Standard attribute keys: | -------------------------- | ---------------------------------------------------------- | | `hyperdx.team.id` | Owning team (multi-tenancy boundary). Set this everywhere. | | `user.id` | Acting user (OTel `user.*` semconv). | -| `user.email` | Acting user's email (OTel `user.*` semconv). | +| `user.email` | Acting user's email (OTel `user.*` semconv). | | `hyperdx..` | Domain IDs, e.g. `hyperdx.alert.id`, `hyperdx.source.id`. | | `feature_flag.` | Evaluated feature/config flag state. | @@ -124,8 +125,8 @@ Standard attribute keys: > **Scope:** this is ClickStack's _internal_ instrumentation philosophy for the > code in this repo — not a recommendation that users pick wide events over -> metrics. Metrics are a first-class ClickStack signal that many operators rely on; -> nothing here discourages them. What follows is simply how _we_ prefer to +> metrics. Metrics are a first-class ClickStack signal that many operators rely +> on; nothing here discourages them. What follows is simply how _we_ prefer to > instrument our own services. We instrument in the spirit of @@ -214,10 +215,10 @@ To make a piece of functionality SLO-able, wrap it with `withOperationMetrics` a streaming proxy). Both emit a standard pair of SLI signals tagged with a stable, low-cardinality `operation` name and `outcome` (`success` | `error`): -| Metric | Type | Use as | -| ------------------------------- | --------- | --------------- | +| Metric | Type | Use as | +| ------------------------------- | --------- | ---------------- | | `hyperdx.operation.requests` | counter | availability SLI | -| `hyperdx.operation.duration_ms` | histogram | latency SLI | +| `hyperdx.operation.duration_ms` | histogram | latency SLI | ```ts import { withOperationMetrics } from '@/utils/instrumentation'; @@ -230,8 +231,8 @@ const chartConfig = await withOperationMetrics( ``` Because every operation reports through the same two metrics, an SLO is just a -filter on `operation` — availability = `outcome:success` / total, latency = -the duration histogram. Keep `operation` a constant (e.g. `ai.assistant`, +filter on `operation` — availability = `outcome:success` / total, latency = the +duration histogram. Keep `operation` a constant (e.g. `ai.assistant`, `clickhouse_proxy.query`); never interpolate IDs or user input into it. Reach for this on functionality with real failure modes worth a target (external dependencies, query proxies, AI calls) — not on thin CRUD that the HTTP @@ -239,18 +240,18 @@ auto-instrumentation and error middleware already cover. ## Query attribution -Every ClickHouse query ClickStack runs is tagged with what asked for it, so a row -in the customer's `system.query_log` points back to a dashboard tile, a saved -search, an alert, or an MCP tool. This is the one piece of our instrumentation -that lands in *their* data rather than ours, so it stays small, carries no -identity, and must never be able to fail a query. +Every ClickHouse query ClickStack runs is tagged with what asked for it, so a +row in the customer's `system.query_log` points back to a dashboard tile, a +saved search, an alert, or an MCP tool. This is the one piece of our +instrumentation that lands in _their_ data rather than ours, so it stays small, +carries no identity, and must never be able to fail a query. Two channels carry it, set by `BaseClickhouseClient.query`: -| Where | Shape | What it answers | -| --- | --- | --- | +| Where | Shape | What it answers | +| ------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `log_comment` | JSON payload, e.g. `{"v":1,"surface":"dashboard","dashboard":"…","tile":"…"}` | After the fact: `GROUP BY JSONExtractString(log_comment, 'tile')` over `system.query_log` | -| `query_id` | `hdx--` | Right now: reading `system.processes` while a query is still running | +| `query_id` | `hdx--` | Right now: reading `system.processes` while a query is still running | The vocabulary lives in [`packages/common-utils/src/clickhouse/attribution.ts`](../packages/common-utils/src/clickhouse/attribution.ts). @@ -304,14 +305,13 @@ calls `/api/v1/prometheus/query{,_range}`, and `/labels` and `buildLogComment(attribution)` in the `x-hyperdx-query-attribution` header (`QUERY_ATTRIBUTION_HEADER`), and the router parses it with `parseLogComment` into a `withQueryAttribution` scope layered over the auth middleware's `api` -scope. The browser's surface and ids win; `trace` and `label` stay the -server's. Queries the route runs through a ClickHouse client (label lookups -before 26.9, and the `prometheusQuery()`/`prometheusQueryRange()` fallback) -pick the scope up as usual. Requests proxied to ClickHouse's Prometheus HTTP -API (queries on 26.6+, label lookups on 26.9+) are tagged -by `clickhousePrometheusRequest`: `log_comment` as a URL setting, `query_id` as -the `X-ClickHouse-Query-Id` header, because the handler rejects `query_id` as an -unknown setting in the URL. +scope. The browser's surface and ids win; `trace` and `label` stay the server's. +Queries the route runs through a ClickHouse client (label lookups before 26.9, +and the `prometheusQuery()`/`prometheusQueryRange()` fallback) pick the scope up +as usual. Requests proxied to ClickHouse's Prometheus HTTP API (queries on +26.6+, label lookups on 26.9+) are tagged by `clickhousePrometheusRequest`: +`log_comment` as a URL setting, `query_id` as the `X-ClickHouse-Query-Id` +header, because the handler rejects `query_id` as an unknown setting in the URL. ### Rules @@ -356,7 +356,7 @@ unknown setting in the URL. - SLO operation metrics (`withOperationMetrics`) on an external dependency: `packages/api/src/routers/api/ai.ts` - SLO operation metrics (`recordOperationOutcome`) on a streaming proxy: - `packages/api/src/routers/api/clickhouseProxy.ts` + `packages/api/src/utils/proxyOutcome.ts` - End-to-end + sub-operation SLO metrics on a background job (alert evaluation and its data fetch): `packages/api/src/tasks/checkAlerts/index.ts` (`alerts.evaluate`, `alerts.query`) diff --git a/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts b/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts new file mode 100644 index 000000000..6bbd42512 --- /dev/null +++ b/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts @@ -0,0 +1,239 @@ +import http from 'http'; + +import { getLoggedInAgent, getServer } from '@/fixtures'; +import Connection from '@/models/connection'; +import * as instrumentation from '@/utils/instrumentation'; + +jest.unmock('http-proxy-middleware'); + +const TIMEOUT_TRAILER = + 'Code: 159. DB::Exception: Timeout exceeded: elapsed 15.0 seconds, maximum: 15. (TIMEOUT_EXCEEDED) (version 25.3.1.1)\n'; + +// Each scenario is what ClickHouse (or the endpoint in front of it) does for +// the query id in the request. +const scenarios: Record< + string, + (req: http.IncomingMessage, res: http.ServerResponse) => void +> = { + ok: (_req, res) => { + res.writeHead(200); + res.end('{"a":1}\n'); + }, + 'mid-stream': (_req, res) => { + res.writeHead(200); + res.write('{"a":1}\n'); + res.end(TIMEOUT_TRAILER); + }, + tagged: (_req, res) => { + const tag = 'abcdefghijklmnop'; + res.writeHead(200, { 'X-ClickHouse-Exception-Tag': tag }); + res.write('{"a":1}\n'); + res.end( + `\r\n__exception__\r\n${tag}\r\n${TIMEOUT_TRAILER}${Buffer.byteLength(TIMEOUT_TRAILER)} ${tag}\r\n__exception__\r\n`, + ); + }, + overloaded: (_req, res) => { + res.writeHead(500, { 'X-ClickHouse-Exception-Code': '202' }); + res.end( + 'Code: 202. DB::Exception: Too many simultaneous queries. (TOO_MANY_SIMULTANEOUS_QUERIES)\n', + ); + }, + unavailable: (_req, res) => { + res.writeHead(503); + res.end('Service is starting up'); + }, + cut: (req, res) => { + res.writeHead(200); + res.write('{"a":1}\n', () => req.socket.destroy()); + }, + hang: () => {}, +}; + +const listen = async (server: http.Server) => { + await new Promise(resolve => server.listen(0, resolve)); + const address = server.address(); + if (address == null || typeof address === 'string') { + throw new Error('Expected a TCP port'); + } + return `http://127.0.0.1:${address.port}`; +}; + +describe('clickhouse-proxy outcome', () => { + const server = getServer(); + let upstream: http.Server; + let upstreamUrl: string; + let recordSpy: jest.SpyInstance; + const span = { setAttributes: jest.fn() }; + + beforeAll(async () => { + await server.start(); + upstream = http.createServer((req, res) => { + req.resume(); + req.on('end', () => { + const id = new URL(req.url ?? '', 'http://x').searchParams.get( + 'query_id', + ); + scenarios[id ?? 'ok'](req, res); + }); + }); + upstreamUrl = await listen(upstream); + }); + + beforeEach(() => { + recordSpy = jest.spyOn(instrumentation, 'recordOperationOutcome'); + jest + .spyOn(instrumentation, 'getServerSpan') + .mockReturnValue(span as unknown as instrumentation.Span); + }); + + afterEach(async () => { + jest.restoreAllMocks(); + span.setAttributes.mockReset(); + await server.clearDBs(); + }); + + afterAll(async () => { + upstream.closeAllConnections(); + await new Promise(resolve => upstream.close(() => resolve())); + await server.stop(); + }); + + // Returns a request builder, not a request: awaiting a supertest request + // would send it before the caller could add assertions. + const connect = async (host = upstreamUrl) => { + const { agent, team } = await getLoggedInAgent(server); + const connection = await Connection.create({ + team: team._id, + name: 'Scripted upstream', + host, + username: 'default', + password: '', + }); + return (queryId: string) => + agent + .post(`/clickhouse-proxy/?query_id=${queryId}`) + .set('x-hyperdx-connection-id', connection._id.toString()) + .set('Content-Type', 'text/plain') + .send('SELECT 1'); + }; + + const proxyOutcome = async () => { + const find = () => + recordSpy.mock.calls + .map(([args]) => args) + .find(args => args.operation === 'clickhouse_proxy.query'); + for (let i = 0; i < 50 && !find(); i++) { + await new Promise(resolve => setTimeout(resolve, 20)); + } + return find(); + }; + + it('leaves a clean response uncategorised', async () => { + const send = await connect(); + await send('ok').expect(200); + expect(await proxyOutcome()).toMatchObject({ + outcome: 'success', + attributes: undefined, + }); + }); + + it('categorises an exception written after a 200', async () => { + const send = await connect(); + const res = await send('mid-stream') + .buffer(true) + .parse((r, cb) => { + let text = ''; + r.on('data', (chunk: Buffer) => (text += chunk)); + r.on('end', () => cb(null, text)); + }); + expect(res.status).toBe(200); + expect(res.body).toBe(`{"a":1}\n${TIMEOUT_TRAILER}`); + expect(await proxyOutcome()).toMatchObject({ + outcome: 'success', + attributes: { error_category: 'query_timeout' }, + }); + expect(span.setAttributes).toHaveBeenCalledWith( + expect.objectContaining({ + 'hyperdx.clickhouse_proxy.clickhouse.error_code': 159, + 'hyperdx.clickhouse_proxy.upstream.status_code': 200, + }), + ); + }); + + it('categorises a tagged exception written after a 200', async () => { + const send = await connect(); + await send('tagged').expect(200); + expect(await proxyOutcome()).toMatchObject({ + attributes: { error_category: 'query_timeout' }, + }); + }); + + it('reads the ClickHouse error code header', async () => { + const send = await connect(); + await send('overloaded').expect(500); + expect(await proxyOutcome()).toMatchObject({ + outcome: 'error', + attributes: { error_category: 'service_overloaded' }, + }); + }); + + it('records the body of an upstream 5xx', async () => { + const send = await connect(); + await send('unavailable').expect(503); + expect(await proxyOutcome()).toMatchObject({ + outcome: 'error', + attributes: { error_category: 'upstream_unavailable' }, + }); + expect(span.setAttributes).toHaveBeenCalledWith( + expect.objectContaining({ + 'hyperdx.clickhouse_proxy.error.message': 'Service is starting up', + }), + ); + }); + + it('categorises a stream the upstream cut off', async () => { + const send = await connect(); + await expect(send('cut')).rejects.toThrow(); + expect(await proxyOutcome()).toMatchObject({ + attributes: { error_category: 'upstream_unavailable' }, + }); + }); + + it('categorises a refused connection', async () => { + const closed = http.createServer(); + const closedUrl = await listen(closed); + await new Promise(resolve => closed.close(() => resolve())); + + const send = await connect(closedUrl); + await send('ok').expect(500); + expect(await proxyOutcome()).toMatchObject({ + outcome: 'error', + attributes: { error_category: 'upstream_unavailable' }, + }); + }); + + it('categorises a request the client abandoned', async () => { + const send = await connect(); + const request = send('hang'); + void request.then( + () => {}, + () => {}, + ); + // Close the socket the way a browser does when a search is re-run. + await new Promise(resolve => setTimeout(resolve, 100)); + // superagent's abort() leaves the agent's socket open. + if ('req' in request && request.req instanceof http.ClientRequest) { + request.req.destroy(); + } + for (let i = 0; i < 50 && !span.setAttributes.mock.calls.length; i++) { + await new Promise(resolve => setTimeout(resolve, 20)); + } + expect(span.setAttributes).toHaveBeenCalledWith( + expect.objectContaining({ + 'hyperdx.clickhouse_proxy.error.category': 'client_aborted', + }), + ); + // Nothing was answered, so the SLI has nothing to count. + expect(await proxyOutcome()).toBeUndefined(); + }); +}); diff --git a/packages/api/src/routers/api/clickhouseProxy.ts b/packages/api/src/routers/api/clickhouseProxy.ts index d67e00b11..3902e7e07 100644 --- a/packages/api/src/routers/api/clickhouseProxy.ts +++ b/packages/api/src/routers/api/clickhouseProxy.ts @@ -12,15 +12,19 @@ import { validateRequestHeaders } from '@/middleware/validation'; import { recordOperationOutcome } from '@/utils/instrumentation'; import logger from '@/utils/logger'; import { writeProxyBody } from '@/utils/proxyBody'; +import { + noteTransportError, + observeProxyRequest, + observeUpstreamResponse, + trackProxyOutcome, +} from '@/utils/proxyOutcome'; import { IPV6_BRACKET_RE, isPrivateIp } from '@/utils/validators'; import { objectIdSchema } from '@/utils/zod'; -// SLO operations for the ClickHouse proxy. Both paths swallow their errors -// (returning JSON / writing the response directly) so they never reach the API -// error middleware — they must report their own SLIs. See -// agent_docs/observability.md. +// The connection test swallows its errors (returning JSON) so they never reach +// the API error middleware; it must report its own SLI. The query proxy's is +// recorded by trackProxyOutcome. See agent_docs/observability.md. const CONNECTION_TEST_OPERATION = 'clickhouse_proxy.connection_test'; -const QUERY_PROXY_OPERATION = 'clickhouse_proxy.query'; /** * Validates and sanitizes a URL path to prevent injection attacks. @@ -243,6 +247,7 @@ const proxyMiddleware: RequestHandler = }, on: { proxyReq: (proxyReq, _req, res) => { + observeProxyRequest(proxyReq, res); // set user-agent to the hyperdx version identifier proxyReq.setHeader('user-agent', `hyperdx ${CODE_VERSION}`); @@ -265,16 +270,7 @@ const proxyMiddleware: RequestHandler = writeProxyBody(proxyReq, _req); }, proxyRes: (proxyRes, _req, res) => { - const startedAt = (res as Response).locals?.hdxProxyStartedAt; - const statusCode = proxyRes.statusCode ?? 0; - recordOperationOutcome({ - operation: QUERY_PROXY_OPERATION, - // A response (even a 4xx/5xx from ClickHouse) means the proxy hop - // itself worked; outcome reflects whether ClickHouse served it. - outcome: statusCode < 400 ? 'success' : 'error', - durationMs: - typeof startedAt === 'number' ? performance.now() - startedAt : 0, - }); + observeUpstreamResponse(proxyRes, res); // since clickhouse v24, the cors headers * will be attached to the response by default // which will cause the browser to block the response @@ -294,20 +290,19 @@ const proxyMiddleware: RequestHandler = } }, error: (err, _req, _res) => { - const startedAt = (_res as Response).locals?.hdxProxyStartedAt; - recordOperationOutcome({ - operation: QUERY_PROXY_OPERATION, - // No usable response from ClickHouse (connection refused, timeout, - // DNS failure, ...) — a hard availability failure for the proxy. - outcome: 'error', - durationMs: - typeof startedAt === 'number' ? performance.now() - startedAt : 0, - }); + const res = _res as Response; + noteTransportError(res, err); console.error('Proxy error:', err); - (_res as Response).writeHead(500, { + // The upstream broke off mid-stream: a second status would throw, and + // ending cleanly would pass the truncated rows off as complete. + if (res.headersSent) { + res.destroy(); + return; + } + res.writeHead(500, { 'Content-Type': 'application/json', }); - _res.end( + res.end( JSON.stringify({ success: false, error: err.message || 'Failed to connect to ClickHouse server', @@ -320,24 +315,18 @@ const proxyMiddleware: RequestHandler = // }), }); -// Stamp a start time so the proxy callbacks can record query SLO latency. -const markProxyStart: RequestHandler = (_req, res, next) => { - res.locals.hdxProxyStartedAt = performance.now(); - next(); -}; - router.get( '/*', hasConnectionId, getConnection, - markProxyStart, + trackProxyOutcome, proxyMiddleware, ); router.post( '/*', hasConnectionId, getConnection, - markProxyStart, + trackProxyOutcome, proxyMiddleware, ); diff --git a/packages/api/src/utils/__tests__/proxyOutcome.test.ts b/packages/api/src/utils/__tests__/proxyOutcome.test.ts new file mode 100644 index 000000000..1c2fd84cb --- /dev/null +++ b/packages/api/src/utils/__tests__/proxyOutcome.test.ts @@ -0,0 +1,107 @@ +import { + classifyProxyFailure, + findTrailingException, + ProxyFacts, +} from '@/utils/proxyOutcome'; + +const ok: ProxyFacts = { + clientAborted: false, + timedOut: false, + upstreamCut: false, + upstreamStatus: 200, + responseStatus: 200, +}; + +describe('classifyProxyFailure', () => { + it.each<[string, Partial, string | undefined]>([ + ['a clean 200', {}, undefined], + ['TIMEOUT_EXCEEDED', { clickhouseErrorCode: 159 }, 'query_timeout'], + [ + 'CANNOT_SCHEDULE_TASK', + { clickhouseErrorCode: 439 }, + 'service_overloaded', + ], + [ + 'TOO_MANY_SIMULTANEOUS_QUERIES', + { upstreamStatus: 500, clickhouseErrorCode: 202 }, + 'service_overloaded', + ], + [ + 'UNKNOWN_TABLE', + { upstreamStatus: 404, clickhouseErrorCode: 60 }, + 'query_error', + ], + [ + 'the proxy timeout', + { + timedOut: true, + upstreamStatus: undefined, + transportError: { code: 'ECONNRESET' }, + }, + 'upstream_timeout', + ], + [ + 'a refused connection', + { upstreamStatus: undefined, transportError: { code: 'ECONNREFUSED' } }, + 'upstream_unavailable', + ], + [ + 'a transport error without an errno', + { upstreamStatus: undefined, transportError: {} }, + 'proxy_error', + ], + ['a stream cut after a 200', { upstreamCut: true }, 'upstream_unavailable'], + [ + 'a 503 with no ClickHouse code', + { upstreamStatus: 503 }, + 'upstream_unavailable', + ], + ['a 504 from the endpoint', { upstreamStatus: 504 }, 'upstream_timeout'], + ['a 429 from the endpoint', { upstreamStatus: 429 }, 'service_overloaded'], + ['a 403 with no ClickHouse code', { upstreamStatus: 403 }, 'query_error'], + [ + 'a client abort', + { clientAborted: true, clickhouseErrorCode: 394 }, + 'client_aborted', + ], + [ + 'a proxy failure before any upstream request', + { upstreamStatus: undefined, responseStatus: 500 }, + 'proxy_error', + ], + ])('%s', (_name, facts, expected) => { + expect(classifyProxyFailure({ ...ok, ...facts })).toBe(expected); + }); +}); + +describe('findTrailingException', () => { + const timeout = + 'Code: 159. DB::Exception: Timeout exceeded: elapsed 15.0 seconds, maximum: 15. (TIMEOUT_EXCEEDED) (version 25.3.1.1)'; + + it('reads an unmarked exception on the last line', () => { + const body = Buffer.from(`{"a":1}\n{"a":2}\n${timeout}\n`); + expect(findTrailingException(body)).toEqual({ + code: 159, + message: expect.stringContaining('Timeout exceeded'), + }); + }); + + it('reads an exception framed by the exception tag', () => { + const tag = 'abcdefghijklmnop'; + const message = `${timeout}\n`; + const body = Buffer.from( + `{"a":1}\r\n__exception__\r\n${tag}\r\n${message}${Buffer.byteLength(message)} ${tag}\r\n__exception__\r\n`, + ); + expect(findTrailingException(body, tag)?.code).toBe(159); + }); + + it('ignores row data that quotes a ClickHouse error', () => { + const body = Buffer.from(`{"Body":${JSON.stringify(timeout)}}\n`); + expect(findTrailingException(body)).toBeUndefined(); + }); + + it('ignores an untagged trailer when the response carries a tag', () => { + const body = Buffer.from(`{"a":1}\n${timeout}\n`); + expect(findTrailingException(body, 'abcdefghijklmnop')).toBeUndefined(); + }); +}); diff --git a/packages/api/src/utils/instrumentation.ts b/packages/api/src/utils/instrumentation.ts index 39e8e3ff0..c5ec68af1 100644 --- a/packages/api/src/utils/instrumentation.ts +++ b/packages/api/src/utils/instrumentation.ts @@ -143,14 +143,20 @@ export function setBusinessContext(context: BusinessContext): void { const activeSpan = opentelemetry.trace.getActiveSpan(); activeSpan?.setAttributes(attributes); - // The HTTP instrumentation keeps the server span in the request context's - // RPC metadata. There is none outside a request, as in scheduled tasks. - const serverSpan = getRPCMetadata(opentelemetry.context.active())?.span; + const serverSpan = getServerSpan(); if (serverSpan && serverSpan !== activeSpan && serverSpan.isRecording()) { serverSpan.setAttributes(attributes); } } +/** + * The current request's HTTP server span, which the HTTP instrumentation keeps + * in the context's RPC metadata. None outside a request, as in scheduled tasks. + */ +export function getServerSpan(): Span | undefined { + return getRPCMetadata(opentelemetry.context.active())?.span; +} + /** * Trace id of the span in flight, if any. Tagged onto ClickHouse queries so a * row in `system.query_log` links back to the request that ran it. diff --git a/packages/api/src/utils/proxyOutcome.ts b/packages/api/src/utils/proxyOutcome.ts new file mode 100644 index 000000000..d47511e24 --- /dev/null +++ b/packages/api/src/utils/proxyOutcome.ts @@ -0,0 +1,270 @@ +import { + EXCEPTION_TAG_HEADER_NAME, + extractErrorAtTheEndOfChunk, + parseError, +} from '@clickhouse/client-common'; +import { errorMonitor } from 'events'; +import type { RequestHandler } from 'express'; +import type { ClientRequest, IncomingMessage, ServerResponse } from 'http'; +import { performance } from 'perf_hooks'; + +import { isClickHouseError } from '@/utils/clickhouseErrors'; +import { + getServerSpan, + recordOperationOutcome, + type Span, +} from '@/utils/instrumentation'; + +const QUERY_PROXY_OPERATION = 'clickhouse_proxy.query'; + +/** + * Why a proxied ClickHouse request failed. Recorded on the server span and as + * the `error_category` attribute of the `clickhouse_proxy.query` SLI, so the + * SLO can choose which categories count against it (HDX-6072). + */ +export type ProxyErrorCategory = + | 'upstream_timeout' + | 'upstream_unavailable' + | 'service_overloaded' + | 'query_timeout' + | 'query_error' + | 'client_aborted' + | 'proxy_error'; + +export type ProxyFacts = { + clientAborted: boolean; + /** The proxy's own upstream timeout fired. */ + timedOut: boolean; + /** No usable response: set from the proxy `error` event. */ + transportError?: { code?: string }; + upstreamStatus?: number; + /** The upstream response body ended before it was complete. */ + upstreamCut: boolean; + clickhouseErrorCode?: number; + responseStatus: number; +}; + +// ClickHouse error codes, from src/Common/ErrorCodes.cpp. +const QUERY_TIMEOUT_CODES = new Set([ + 159, // TIMEOUT_EXCEEDED + 160, // TOO_SLOW +]); +const OVERLOADED_CODES = new Set([ + 202, // TOO_MANY_SIMULTANEOUS_QUERIES + 203, // NO_FREE_CONNECTION + 439, // CANNOT_SCHEDULE_TASK +]); +const UNREACHABLE_ERRNOS = new Set([ + 'ECONNREFUSED', + 'ECONNRESET', + 'EPIPE', + 'ENOTFOUND', + 'EAI_AGAIN', + 'EHOSTUNREACH', + 'ENETUNREACH', +]); + +export function classifyProxyFailure( + f: ProxyFacts, +): ProxyErrorCategory | undefined { + if (f.clientAborted) return 'client_aborted'; + const code = f.clickhouseErrorCode; + if (code != null) { + if (QUERY_TIMEOUT_CODES.has(code)) return 'query_timeout'; + if (OVERLOADED_CODES.has(code)) return 'service_overloaded'; + return 'query_error'; + } + if (f.timedOut || f.transportError?.code === 'ETIMEDOUT') { + return 'upstream_timeout'; + } + if (f.upstreamCut) return 'upstream_unavailable'; + if (f.transportError) { + return UNREACHABLE_ERRNOS.has(f.transportError.code ?? '') + ? 'upstream_unavailable' + : 'proxy_error'; + } + const status = f.upstreamStatus; + if (status != null) { + if (status === 504) return 'upstream_timeout'; + if (status === 429) return 'service_overloaded'; + if (status >= 500) return 'upstream_unavailable'; + if (status >= 400) return 'query_error'; + return undefined; + } + return f.responseStatus >= 400 ? 'proxy_error' : undefined; +} + +const EXCEPTION_MARKER = '__exception__'; + +/** + * Finds a ClickHouse exception written at the end of a response body, which is + * how a query fails after it has started streaming (`wait_end_of_query=0`). + * Since 25.11 the exception is framed with a per-response tag from the + * `X-ClickHouse-Exception-Tag` header; before that it is an unmarked last line. + */ +export function findTrailingException( + tail: Buffer, + exceptionTag?: string, +): { code?: number; message: string } | undefined { + if (exceptionTag) { + const trailer = `${exceptionTag}\r\n${EXCEPTION_MARKER}\r\n`; + if (!tail.toString('latin1').endsWith(trailer)) return undefined; + return toException(extractErrorAtTheEndOfChunk(tail, exceptionTag)); + } + const lastLine = tail.toString('utf8').trimEnd().split('\n').pop() ?? ''; + if (!/^Code: \d+\. DB::Exception/.test(lastLine)) return undefined; + return toException(parseError(lastLine)); +} + +function toException(err: Error): { code?: number; message: string } { + const code = isClickHouseError(err) ? Number(err.code) : NaN; + return { + code: Number.isFinite(code) ? code : undefined, + message: err.message, + }; +} + +// Enough to hold a ClickHouse exception, which is all the tail is read for. +const TAIL_BYTES = 8 * 1024; +const MAX_MESSAGE_LENGTH = 1000; + +type ProxyState = { + startedAt: number; + headersAt?: number; + span?: Span; + timedOut: boolean; + upstreamCut: boolean; + transportError?: { code?: string; message: string }; + upstreamStatus?: number; + headerErrorCode?: number; + exceptionTag?: string; + tail?: Buffer; + done: boolean; +}; + +const states = new WeakMap(); + +/** + * Starts tracking one proxied request. Its outcome is recorded once, when the + * response closes, from listeners prepended so they run before the HTTP + * instrumentation ends the server span. + */ +export const trackProxyOutcome: RequestHandler = (_req, res, next) => { + const state: ProxyState = { + startedAt: performance.now(), + span: getServerSpan(), + timedOut: false, + upstreamCut: false, + done: false, + }; + states.set(res, state); + const finish = () => finishProxyOutcome(res, state); + res.prependListener('close', finish); + res.prependListener(errorMonitor, finish); + next(); +}; + +export function observeProxyRequest( + proxyReq: ClientRequest, + res: ServerResponse, +) { + const state = states.get(res); + if (!state) return; + proxyReq.once('timeout', () => { + state.timedOut = true; + }); +} + +export function observeUpstreamResponse( + proxyRes: IncomingMessage, + res: ServerResponse, +) { + const state = states.get(res); + if (!state) return; + state.headersAt = performance.now(); + state.upstreamStatus = proxyRes.statusCode; + const headerCode = Number(proxyRes.headers['x-clickhouse-exception-code']); + state.headerErrorCode = Number.isFinite(headerCode) ? headerCode : undefined; + const tag = proxyRes.headers[EXCEPTION_TAG_HEADER_NAME]; + state.exceptionTag = typeof tag === 'string' ? tag : undefined; + + // These run before the proxy's own listeners, which destroy the response. + const markCut = () => { + if (!proxyRes.complete) state.upstreamCut = true; + }; + proxyRes.once('aborted', markCut); + proxyRes.once('error', markCut); + + // shortcut: a compressed body is not inspected, so an exception at its end + // goes undetected; decompress the tail if browsers start asking for gzip. + if (proxyRes.headers['content-encoding']) return; + proxyRes.on('data', (chunk: Buffer) => { + state.tail = + chunk.length >= TAIL_BYTES || !state.tail + ? chunk.subarray(-TAIL_BYTES) + : Buffer.concat([state.tail, chunk]).subarray(-TAIL_BYTES); + }); +} + +export function noteTransportError(res: ServerResponse, err: Error) { + const state = states.get(res); + if (!state) return; + const code = + 'code' in err && typeof err.code === 'string' ? err.code : undefined; + state.transportError = { code, message: err.message }; +} + +function finishProxyOutcome(res: ServerResponse, state: ProxyState) { + if (state.done) return; + state.done = true; + + const failedUpstream = + state.timedOut || state.upstreamCut || state.transportError != null; + const isErrorStatus = (state.upstreamStatus ?? 0) >= 400; + const exception = state.tail + ? findTrailingException(state.tail, state.exceptionTag) + : undefined; + const clickhouseErrorCode = state.headerErrorCode ?? exception?.code; + + const category = classifyProxyFailure({ + clientAborted: !res.writableFinished && !failedUpstream, + timedOut: state.timedOut, + transportError: state.transportError, + upstreamStatus: state.upstreamStatus, + upstreamCut: state.upstreamCut, + clickhouseErrorCode, + responseStatus: res.statusCode, + }); + + const message = + exception?.message ?? + state.transportError?.message ?? + (isErrorStatus ? state.tail?.toString('utf8') : undefined); + + state.span?.setAttributes({ + 'hyperdx.clickhouse_proxy.error.category': category, + 'hyperdx.clickhouse_proxy.upstream.status_code': state.upstreamStatus, + 'hyperdx.clickhouse_proxy.clickhouse.error_code': clickhouseErrorCode, + 'hyperdx.clickhouse_proxy.error.message': category + ? message?.slice(0, MAX_MESSAGE_LENGTH) + : undefined, + }); + + // A request abandoned before ClickHouse answered never had an outcome; the + // SLI has never counted it, and Node's default 200 would call it a success. + if (category === 'client_aborted' && state.headersAt == null) return; + + recordOperationOutcome({ + operation: QUERY_PROXY_OPERATION, + // Outcome keeps its status-based meaning; error_category is what lets the + // SLO decide which failures count. + outcome: + state.transportError || (state.upstreamStatus ?? res.statusCode) >= 400 + ? 'error' + : 'success', + // Time to the upstream's response headers, as before, so the latency SLI + // keeps measuring the same thing. + durationMs: (state.headersAt ?? performance.now()) - state.startedAt, + attributes: category ? { error_category: category } : undefined, + }); +} From 9e6b4700f671fb78a4065bffd5bbe1e9cf2d8282 Mon Sep 17 00:00:00 2001 From: Jordan Simonovski Date: Wed, 18 Nov 2026 17:26:02 +1100 Subject: [PATCH 2/4] fix(api): keep ClickHouse error messages off the proxy span ClickHouse quotes the query and its literals in exception messages, so record the error type instead. Raw bodies are kept only for upstream errors without a ClickHouse code, such as an endpoint's 503 page. --- .../classify-clickhouse-proxy-failures.md | 5 +++-- .../clickhouseProxyOutcome.int.test.ts | 9 +++++++++ .../src/utils/__tests__/proxyOutcome.test.ts | 2 +- packages/api/src/utils/proxyOutcome.ts | 20 ++++++++++--------- 4 files changed, 24 insertions(+), 12 deletions(-) diff --git a/.changeset/classify-clickhouse-proxy-failures.md b/.changeset/classify-clickhouse-proxy-failures.md index 5fbfc2b24..0bb5e8e84 100644 --- a/.changeset/classify-clickhouse-proxy-failures.md +++ b/.changeset/classify-clickhouse-proxy-failures.md @@ -4,5 +4,6 @@ feat: Classify failed ClickHouse proxy requests on the server span and the `clickhouse_proxy.query` SLI (`error_category`), including errors ClickHouse -writes after a 200 response. The span also carries the upstream status, the -ClickHouse error code and the error message. +writes after a 200 response. The span also records the upstream status, the +ClickHouse error code and type, and the body of upstream errors that carry no +ClickHouse code. diff --git a/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts b/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts index 6bbd42512..2eb3df7be 100644 --- a/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts +++ b/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts @@ -155,6 +155,8 @@ describe('clickhouse-proxy outcome', () => { expect(span.setAttributes).toHaveBeenCalledWith( expect.objectContaining({ 'hyperdx.clickhouse_proxy.clickhouse.error_code': 159, + 'hyperdx.clickhouse_proxy.clickhouse.error_type': 'TIMEOUT_EXCEEDED', + 'hyperdx.clickhouse_proxy.error.message': undefined, 'hyperdx.clickhouse_proxy.upstream.status_code': 200, }), ); @@ -175,6 +177,13 @@ describe('clickhouse-proxy outcome', () => { outcome: 'error', attributes: { error_category: 'service_overloaded' }, }); + expect(span.setAttributes).toHaveBeenCalledWith( + expect.objectContaining({ + 'hyperdx.clickhouse_proxy.clickhouse.error_type': + 'TOO_MANY_SIMULTANEOUS_QUERIES', + 'hyperdx.clickhouse_proxy.error.message': undefined, + }), + ); }); it('records the body of an upstream 5xx', async () => { diff --git a/packages/api/src/utils/__tests__/proxyOutcome.test.ts b/packages/api/src/utils/__tests__/proxyOutcome.test.ts index 1c2fd84cb..ead5416de 100644 --- a/packages/api/src/utils/__tests__/proxyOutcome.test.ts +++ b/packages/api/src/utils/__tests__/proxyOutcome.test.ts @@ -82,7 +82,7 @@ describe('findTrailingException', () => { const body = Buffer.from(`{"a":1}\n{"a":2}\n${timeout}\n`); expect(findTrailingException(body)).toEqual({ code: 159, - message: expect.stringContaining('Timeout exceeded'), + type: 'TIMEOUT_EXCEEDED', }); }); diff --git a/packages/api/src/utils/proxyOutcome.ts b/packages/api/src/utils/proxyOutcome.ts index d47511e24..c9afb9c2b 100644 --- a/packages/api/src/utils/proxyOutcome.ts +++ b/packages/api/src/utils/proxyOutcome.ts @@ -105,7 +105,7 @@ const EXCEPTION_MARKER = '__exception__'; export function findTrailingException( tail: Buffer, exceptionTag?: string, -): { code?: number; message: string } | undefined { +): { code?: number; type?: string } | undefined { if (exceptionTag) { const trailer = `${exceptionTag}\r\n${EXCEPTION_MARKER}\r\n`; if (!tail.toString('latin1').endsWith(trailer)) return undefined; @@ -116,12 +116,11 @@ export function findTrailingException( return toException(parseError(lastLine)); } -function toException(err: Error): { code?: number; message: string } { - const code = isClickHouseError(err) ? Number(err.code) : NaN; - return { - code: Number.isFinite(code) ? code : undefined, - message: err.message, - }; +// The message is left out: ClickHouse quotes the query and its literals in it. +function toException(err: Error): { code?: number; type?: string } { + if (!isClickHouseError(err)) return {}; + const code = Number(err.code); + return { code: Number.isFinite(code) ? code : undefined, type: err.type }; } // Enough to hold a ClickHouse exception, which is all the tail is read for. @@ -236,15 +235,18 @@ function finishProxyOutcome(res: ServerResponse, state: ProxyState) { responseStatus: res.statusCode, }); + // Only bodies without a ClickHouse code, such as an endpoint's 503 page. const message = - exception?.message ?? state.transportError?.message ?? - (isErrorStatus ? state.tail?.toString('utf8') : undefined); + (isErrorStatus && clickhouseErrorCode == null + ? state.tail?.toString('utf8') + : undefined); state.span?.setAttributes({ 'hyperdx.clickhouse_proxy.error.category': category, 'hyperdx.clickhouse_proxy.upstream.status_code': state.upstreamStatus, 'hyperdx.clickhouse_proxy.clickhouse.error_code': clickhouseErrorCode, + 'hyperdx.clickhouse_proxy.clickhouse.error_type': exception?.type, 'hyperdx.clickhouse_proxy.error.message': category ? message?.slice(0, MAX_MESSAGE_LENGTH) : undefined, From 066dfac46c90b177c3dfbbd22ab29ae34e59131a Mon Sep 17 00:00:00 2001 From: Jordan Simonovski Date: Wed, 18 Nov 2026 18:05:44 +1100 Subject: [PATCH 3/4] test(api): use a real span in the proxy outcome tests Drops an unsafe type assertion that pushed api lint past its warning budget. --- .../clickhouseProxyOutcome.int.test.ts | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts b/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts index 2eb3df7be..95fdcb927 100644 --- a/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts +++ b/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts @@ -1,3 +1,4 @@ +import { INVALID_SPAN_CONTEXT, trace } from '@opentelemetry/api'; import http from 'http'; import { getLoggedInAgent, getServer } from '@/fixtures'; @@ -63,7 +64,7 @@ describe('clickhouse-proxy outcome', () => { let upstream: http.Server; let upstreamUrl: string; let recordSpy: jest.SpyInstance; - const span = { setAttributes: jest.fn() }; + let setAttributes: jest.SpyInstance; beforeAll(async () => { await server.start(); @@ -81,14 +82,13 @@ describe('clickhouse-proxy outcome', () => { beforeEach(() => { recordSpy = jest.spyOn(instrumentation, 'recordOperationOutcome'); - jest - .spyOn(instrumentation, 'getServerSpan') - .mockReturnValue(span as unknown as instrumentation.Span); + const span = trace.wrapSpanContext(INVALID_SPAN_CONTEXT); + setAttributes = jest.spyOn(span, 'setAttributes'); + jest.spyOn(instrumentation, 'getServerSpan').mockReturnValue(span); }); afterEach(async () => { jest.restoreAllMocks(); - span.setAttributes.mockReset(); await server.clearDBs(); }); @@ -152,7 +152,7 @@ describe('clickhouse-proxy outcome', () => { outcome: 'success', attributes: { error_category: 'query_timeout' }, }); - expect(span.setAttributes).toHaveBeenCalledWith( + expect(setAttributes).toHaveBeenCalledWith( expect.objectContaining({ 'hyperdx.clickhouse_proxy.clickhouse.error_code': 159, 'hyperdx.clickhouse_proxy.clickhouse.error_type': 'TIMEOUT_EXCEEDED', @@ -177,7 +177,7 @@ describe('clickhouse-proxy outcome', () => { outcome: 'error', attributes: { error_category: 'service_overloaded' }, }); - expect(span.setAttributes).toHaveBeenCalledWith( + expect(setAttributes).toHaveBeenCalledWith( expect.objectContaining({ 'hyperdx.clickhouse_proxy.clickhouse.error_type': 'TOO_MANY_SIMULTANEOUS_QUERIES', @@ -193,7 +193,7 @@ describe('clickhouse-proxy outcome', () => { outcome: 'error', attributes: { error_category: 'upstream_unavailable' }, }); - expect(span.setAttributes).toHaveBeenCalledWith( + expect(setAttributes).toHaveBeenCalledWith( expect.objectContaining({ 'hyperdx.clickhouse_proxy.error.message': 'Service is starting up', }), @@ -234,10 +234,10 @@ describe('clickhouse-proxy outcome', () => { if ('req' in request && request.req instanceof http.ClientRequest) { request.req.destroy(); } - for (let i = 0; i < 50 && !span.setAttributes.mock.calls.length; i++) { + for (let i = 0; i < 50 && !setAttributes.mock.calls.length; i++) { await new Promise(resolve => setTimeout(resolve, 20)); } - expect(span.setAttributes).toHaveBeenCalledWith( + expect(setAttributes).toHaveBeenCalledWith( expect.objectContaining({ 'hyperdx.clickhouse_proxy.error.category': 'client_aborted', }), From c4c00bf64671251ddb43890b049f7b056118ad0c Mon Sep 17 00:00:00 2001 From: Jordan Simonovski Date: Wed, 18 Nov 2026 18:07:39 +1100 Subject: [PATCH 4/4] fix(api): count a cut-off stream as an error and record the start of error bodies A 200 the upstream cuts off mid-stream is now an error on the SLI, the idle-socket timeout only counts when the request failed, and the span records the start of an upstream error page rather than its last 8 KB. --- .../clickhouseProxyOutcome.int.test.ts | 7 +++++-- packages/api/src/utils/proxyOutcome.ts | 21 ++++++++++++------- 2 files changed, 19 insertions(+), 9 deletions(-) diff --git a/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts b/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts index 95fdcb927..2e27e8ce9 100644 --- a/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts +++ b/packages/api/src/routers/api/__tests__/clickhouseProxyOutcome.int.test.ts @@ -41,7 +41,7 @@ const scenarios: Record< }, unavailable: (_req, res) => { res.writeHead(503); - res.end('Service is starting up'); + res.end(`Service is starting up${' '.repeat(10_000)}