Repository navigation
fix(api): use log body fallback in MCP and alert queries - #3681
Conversation
|
@ntmspavan is attempting to deploy a commit to the HyperDX Team on Vercel. A member of the Team first needs to authorize it. |
🦋 Changeset detectedLatest commit: 216b544 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Hi @ntmspavan, thanks for the pull request! Before we review code from a first-time contributor, a maintainer vouches for them, and you're not on our list yet. There's nothing else to file and this PR stays open — a maintainer will look at it, usually within a day or two, and vouch for you right here if it's a genuine attempt at the problem. To make that quick, check the PR description has:
More detail in our contributing guide. |
| }); | ||
| expect(result.isError).toBeFalsy(); | ||
| const output = JSON.parse(getFirstText(result)); | ||
| expect(output.summary.totalCount).toBe(1); |
There was a problem hiding this comment.
🔵 minor — Raw SQL tile test checks that a spy was called, not that the query was correct
The test only asserts that queryChartConfig received bodyExpression: 'Body'. It never checks that $__filters with a bare-text filter matches only the body row, so a macro-expansion regression that ignores bodyExpression would still pass. Fix: give the tile a Lucene filter (e.g. via the tile's filters or dashboard variables) and assert the returned rows contain only needle in body.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Added a real ClickHouse assertion: the MCP-hydrated raw-SQL config is executed with a bare-text filter and must return only the body-matching row. Since query_tile has no runtime-filter input, the test applies it at the query-client boundary.
PR Review
2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor 2 posted as inline comment(s) on the changed lines. Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
Deep Review✅ No critical issues found. This is a tightly scoped fix that threads a log source's 🔵 P3 nitpicks (3)
Reviewers (5): correctness, testing, maintainability, kieran-typescript, project-standards. Testing gaps: The no-regression precedence is pinned (MCP No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
|
Added the raw-SQL assertion and missing-field/trace controls; all 168 affected integration tests and lint/type checks pass. The tile-alert and denoise gaps also exist on the base revision, so I’ve left them as follow-ups under the repo’s scope rules, along with the helper refactor. Both Vercel deployments still need team authorization. |
| source.kind === SourceKind.Log || source.kind === SourceKind.Trace | ||
| ? source.implicitColumnExpression | ||
| : undefined, | ||
| bodyExpression: |
There was a problem hiding this comment.
🔵 minor — computeAliasWithClauses rebuilds the source-to-search-config mapping by hand instead of using buildSearchChartConfig
buildSearchChartConfig in packages/common-utils/src/core/searchChartConfig.ts:132 already maps a source to its search config: implicit column, text-index flag, the logs-only bodyExpression fallback and sample weight. The same file already calls it at checkAlerts/index.ts:829, and template.ts:552 uses it for notification samples. This hand-written copy is the reason the alias query had drifted from the sample query in the first place. Build the config with buildSearchChartConfig(source, { select: resolvedSelect, where, whereLanguage, connection: '' }) instead of adding another field to the copy, so the next source field it learns about reaches the alias path automatically.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| connection: source.connection.toString(), | ||
| timestampValueExpression: source.timestampValueExpression, | ||
| implicitColumnExpression: implicitColumn, | ||
| bodyExpression: |
There was a problem hiding this comment.
🔵 minor — The body-fallback projection is now copied by hand into five MCP builders, and other builders still miss it
The diff pastes source.kind === SourceKind.Log ? source.bodyExpression : undefined next to the existing implicitColumnExpression and useTextIndexForImplicitColumn ternaries in three files: helpers.ts:681 and :758, runEventPatterns.ts:171 and eventDeltas.ts:299. That same hand-copying is why denoise.ts:115/135 and the inline-alert builder at checkAlerts/index.ts:768-789 still lack the fallback, and why eventDeltas.ts:285 has never passed useTextIndexForImplicitColumn. Pull the three-field projection out of buildSearchChartConfig (common-utils/src/core/searchChartConfig.ts:150-165) into an exported helper, for example pickImplicitSearchProps(source), and spread it at every call site.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| 'implicitColumnExpression' in source | ||
| ? source.implicitColumnExpression | ||
| : undefined, | ||
| bodyExpression: |
There was a problem hiding this comment.
🔵 minor — Raw SQL bodyExpression hydration in runConfigTile is never used, and the test that covers it injects the filters by hand
bodyExpression is only read by renderFiltersToSql (packages/common-utils/src/core/renderChartConfig.ts:2602), and that function returns early when chartConfig.filters is empty. On this path the config is {...savedConfig, ...sourceFields, dateRange, variables}. Saved raw SQL configs have no filters field (RawSqlBaseChartConfigSchema / SharedChartSettingsSchema in types.ts), and runConfigTile has no filters option, so the new field cannot change any MCP result. The test admits this ("query_tile exposes variables but not runtime dashboard filters") and adds filters to the captured config before re-running it. That checks the common-utils renderer, not MCP behaviour. Either drop this hunk and the test until query_tile accepts filters, or add the filters input in the same PR and test it end to end through clickstack_query_tile.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| : undefined, | ||
| bodyExpression: | ||
| source.kind === SourceKind.Log ? source.bodyExpression : undefined, | ||
| orderBy: stableSampleExpr, |
There was a problem hiding this comment.
🔵 minor — The rule for which source fields drive bare-text search is now copied by hand at six call sites
source.kind === SourceKind.Log ? source.bodyExpression : undefined now appears in eventDeltas.ts, helpers.ts (twice), runEventPatterns.ts and checkAlerts/index.ts. Each copy sits next to its own copy of the implicitColumnExpression and useTextIndexForImplicitColumn picks. buildSearchChartConfig (packages/common-utils/src/core/searchChartConfig.ts:150-165) already encodes the same rule: Log and Trace get the implicit column, only Log gets the body. Pull it into one helper, e.g. pickTextSearchProps(source) next to pickSampleWeightExpressionProps in packages/common-utils/src/types.ts, and use it here, in buildSearchChartConfig, and in the denoise.ts and inline-alert call sites listed as follow-ups. Then the next consumer can't leave the body out again, which is the bug this PR fixes.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
pulpdrew
left a comment
There was a problem hiding this comment.
LGTM! Well scoped and tested, thanks for the contribution
Summary
Closes #2355.
A log source can define
bodyExpressionwithout a separateimplicitColumnExpression. Bare-text Lucene searches already fall back to that body field in the UI, but several MCP and alert query builders omit it, so the same search fails there.This passes the log source's body expression through MCP builder queries, raw SQL tile hydration, pattern sampling/counting and drill-down filters, event-deltas sampling, and alert alias rendering. Notification samples already use the shared search-config builder; the new tests cover that path too.
The existing serializer still decides which field to search. An explicit implicit column keeps priority, and trace sources receive no body fallback. There are no API or schema changes.
How to reproduce
Starting point: a local checkout with the integration-test services (
make dev-int-buildfollowed bymake dev-int), or a fullyarn devstack. This requires the API; the frontend-only demo cannot exercise these paths.On
main:bodyExpression: Bodyand noimplicitColumnExpression.needle, plus another whose ServiceName containsneedlebut whose Body does not.clickstack_searchwithwhere: needle,whereLanguage: lucene, and a time range covering those logs. It fails with “Can not search bare text without an implicit column set.” The pattern/comparison tools and saved-search alert alias rendering have the same gap.On this branch:
make dev-int FILE=bodyExpressionFallback: all 14 cases pass, including rejection when neither search field is configured and unchanged trace behavior.Tests
Run in a Node 22 container with MongoDB 5.0.32 and ClickHouse 26.10.3.7:
yarn lint:fixyarn lint-stagedmake ci-lintmake ci-unitThe integration command used was:
cd packages/api yarn ci:int src/mcp/__tests__/queryTool.int.test.ts \ src/mcp/__tests__/deltasTool.int.test.ts \ src/mcp/__tests__/emergingSignalsTool.int.test.ts \ src/mcp/__tests__/dashboards/queryTile.int.test.ts \ bodyExpressionFallback.int.test.tsRisks and limitations
This changes previously failing bare-text filters on body-only log sources into working body searches in the listed paths. It reuses existing expression rendering and adds no SQL parser or new query endpoint. The raw SQL test executes the MCP-hydrated config with a Lucene filter against ClickHouse at the query-client boundary:
query_tilecurrently exposes variables but no runtime dashboard-filter input. The full integration and browser E2E suites were not run.The pre-existing omissions in tile/inline-alert evaluation and denoise queries remain separate follow-ups. This PR follows the issue's named call sites and the repository's review-scope rules.
How to test on Vercel preview
N/A — API-only change.
References