Skip to content

Commit bfb5c83

Browse files
authored
fix(api): use log body fallback in MCP and alert queries (#3681)
## Summary Closes #2355. A log source can define `bodyExpression` without a separate `implicitColumnExpression`. Bare-text Lucene searches already fall back to that body field in the UI, but several MCP and alert query builders omit it, so the same search fails there. This passes the log source's body expression through MCP builder queries, raw SQL tile hydration, pattern sampling/counting and drill-down filters, event-deltas sampling, and alert alias rendering. Notification samples already use the shared search-config builder; the new tests cover that path too. The existing serializer still decides which field to search. An explicit implicit column keeps priority, and trace sources receive no body fallback. There are no API or schema changes. ### How to reproduce **Starting point:** a local checkout with the integration-test services (`make dev-int-build` followed by `make dev-int`), or a full `yarn dev` stack. This requires the API; the frontend-only demo cannot exercise these paths. **On `main`:** 1. Configure a log source with `bodyExpression: Body` and no `implicitColumnExpression`. 2. Insert one log whose Body contains `needle`, plus another whose ServiceName contains `needle` but whose Body does not. 3. Call `clickstack_search` with `where: needle`, `whereLanguage: lucene`, and a time range covering those logs. It fails with “Can not search bare text without an implicit column set.” The pattern/comparison tools and saved-search alert alias rendering have the same gap. **On this branch:** 1. Repeat the search. It returns the body-matching row. 2. Set the source's implicit column to ServiceName and repeat it. The ServiceName-matching row is returned instead. 3. Run `make dev-int FILE=bodyExpressionFallback`: all 14 cases pass, including rejection when neither search field is configured and unchanged trace behavior. ### Tests Run in a Node 22 container with MongoDB 5.0.32 and ClickHouse 26.10.3.7: - `yarn lint:fix` - `yarn lint-staged` - `make ci-lint` - `make ci-unit` - Affected integration suites: **168 tests passed**, including the 14 fallback cases. These cover absent/empty implicit columns, explicit-column precedence, map-valued body expressions, raw SQL source hydration and filtered rows, pattern counts and drill-downs, both comparison windows, alert aliases, notification samples, rejection without either search field, and unchanged trace behavior. The integration command used was: ```sh cd packages/api yarn ci:int src/mcp/__tests__/queryTool.int.test.ts \ src/mcp/__tests__/deltasTool.int.test.ts \ src/mcp/__tests__/emergingSignalsTool.int.test.ts \ src/mcp/__tests__/dashboards/queryTile.int.test.ts \ bodyExpressionFallback.int.test.ts ``` ### Risks and limitations This changes previously failing bare-text filters on body-only log sources into working body searches in the listed paths. It reuses existing expression rendering and adds no SQL parser or new query endpoint. The raw SQL test executes the MCP-hydrated config with a Lucene filter against ClickHouse at the query-client boundary: `query_tile` currently exposes variables but no runtime dashboard-filter input. The full integration and browser E2E suites were not run. The pre-existing omissions in tile/inline-alert evaluation and denoise queries remain separate follow-ups. This PR follows the issue's named call sites and the repository's review-scope rules. ### How to test on Vercel preview N/A — API-only change. ### References - Issue: #2355 - Original UI fallback: #2354
1 parent b681cec commit bfb5c83

8 files changed

Lines changed: 446 additions & 1 deletion

File tree

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@hyperdx/api': patch
3+
---
4+
5+
Use the log source body expression for bare-text MCP searches, pattern filters,
6+
event deltas, and alert alias queries when no implicit search column is set.
Lines changed: 274 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,274 @@
1+
import { ClickhouseClient } from '@hyperdx/common-utils/dist/clickhouse/node';
2+
import { isRawSqlChartConfig } from '@hyperdx/common-utils/dist/guards';
3+
import { SourceKind } from '@hyperdx/common-utils/dist/types';
4+
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
5+
6+
import * as config from '@/config';
7+
import {
8+
bulkInsertLogs,
9+
DEFAULT_DATABASE,
10+
DEFAULT_LOGS_TABLE,
11+
getLoggedInAgent,
12+
getServer,
13+
} from '@/fixtures';
14+
import Connection from '@/models/connection';
15+
import { LogSource, Source, SourceDocument } from '@/models/source';
16+
17+
import { callTool, createTestClient, getFirstText } from './mcpTestUtils';
18+
19+
describe('log body fallback for bare-text MCP filters', () => {
20+
const server = getServer();
21+
let client: Client;
22+
let source: SourceDocument;
23+
let connectionId: string;
24+
let now: number;
25+
26+
beforeAll(async () => {
27+
await server.start();
28+
});
29+
30+
beforeEach(async () => {
31+
const { team, user } = await getLoggedInAgent(server);
32+
const connection = await Connection.create({
33+
team: team._id,
34+
name: 'Body fallback',
35+
host: config.CLICKHOUSE_HOST,
36+
username: config.CLICKHOUSE_USER,
37+
password: config.CLICKHOUSE_PASSWORD,
38+
});
39+
connectionId = connection.id;
40+
source = await Source.create({
41+
kind: SourceKind.Log,
42+
team: team._id,
43+
connection: connection._id,
44+
name: 'Logs',
45+
from: { databaseName: DEFAULT_DATABASE, tableName: DEFAULT_LOGS_TABLE },
46+
timestampValueExpression: 'Timestamp',
47+
bodyExpression: 'Body',
48+
});
49+
client = await createTestClient({
50+
teamId: team._id.toString(),
51+
userId: user._id.toString(),
52+
});
53+
now = Date.now();
54+
await bulkInsertLogs(
55+
[2, 7].flatMap(minutes => [
56+
{
57+
Body: 'needle in body',
58+
ServiceName: 'api',
59+
SeverityText: 'error',
60+
LogAttributes: { message: 'other' },
61+
Timestamp: new Date(now - minutes * 60_000),
62+
},
63+
{
64+
Body: 'other',
65+
ServiceName: 'needle',
66+
SeverityText: 'info',
67+
LogAttributes: { message: 'needle in map' },
68+
Timestamp: new Date(now - minutes * 60_000),
69+
},
70+
]),
71+
);
72+
});
73+
74+
afterEach(async () => {
75+
await client.close();
76+
await server.clearDBs();
77+
});
78+
79+
afterAll(async () => {
80+
await server.stop();
81+
});
82+
83+
const currentWindow = () => ({
84+
startTime: new Date(now - 5 * 60_000).toISOString(),
85+
endTime: new Date(now).toISOString(),
86+
});
87+
88+
it.each([
89+
{ implicitColumnExpression: undefined, expectedBody: 'needle in body' },
90+
{ implicitColumnExpression: '', expectedBody: 'needle in body' },
91+
{ implicitColumnExpression: 'ServiceName', expectedBody: 'other' },
92+
])(
93+
'searches the correct field with implicitColumnExpression=$implicitColumnExpression',
94+
async ({ implicitColumnExpression, expectedBody }) => {
95+
await LogSource.updateOne(
96+
{ _id: source._id },
97+
implicitColumnExpression === undefined
98+
? { $unset: { implicitColumnExpression: '' } }
99+
: { $set: { implicitColumnExpression } },
100+
);
101+
const result = await callTool(client, 'clickstack_search', {
102+
sourceId: source._id.toString(),
103+
where: 'needle',
104+
...currentWindow(),
105+
});
106+
expect(result.isError).toBeFalsy();
107+
const output = JSON.parse(getFirstText(result));
108+
expect(output.result.data).toEqual([
109+
expect.objectContaining({ Body: expectedBody }),
110+
]);
111+
},
112+
);
113+
114+
it('uses a map-valued body expression for bare-text search', async () => {
115+
await LogSource.updateOne(
116+
{ _id: source._id },
117+
{ $set: { bodyExpression: "LogAttributes['message']" } },
118+
);
119+
const result = await callTool(client, 'clickstack_search', {
120+
sourceId: source._id.toString(),
121+
where: 'needle',
122+
...currentWindow(),
123+
});
124+
expect(result.isError).toBeFalsy();
125+
expect(JSON.parse(getFirstText(result)).result.data).toEqual([
126+
expect.objectContaining({ Body: 'other' }),
127+
]);
128+
});
129+
130+
it('rejects bare-text search without an implicit column or body expression', async () => {
131+
await LogSource.updateOne(
132+
{ _id: source._id },
133+
{ $unset: { implicitColumnExpression: '', bodyExpression: '' } },
134+
);
135+
const result = await callTool(client, 'clickstack_search', {
136+
sourceId: source._id.toString(),
137+
where: 'needle',
138+
...currentWindow(),
139+
});
140+
expect(result.isError).toBe(true);
141+
expect(getFirstText(result)).toContain(
142+
'Can not search bare text without an implicit column set.',
143+
);
144+
});
145+
146+
it('does not introduce a body fallback for MCP trace searches', async () => {
147+
const traceSource = await Source.create({
148+
kind: SourceKind.Trace,
149+
team: source.team,
150+
connection: source.connection,
151+
name: 'Traces',
152+
from: { databaseName: 'default', tableName: 'otel_traces' },
153+
timestampValueExpression: 'Timestamp',
154+
spanNameExpression: 'SpanName',
155+
});
156+
const result = await callTool(client, 'clickstack_search', {
157+
sourceId: traceSource._id.toString(),
158+
where: 'needle',
159+
...currentWindow(),
160+
});
161+
expect(result.isError).toBe(true);
162+
expect(getFirstText(result)).toContain(
163+
'Can not search bare text without an implicit column set.',
164+
);
165+
});
166+
167+
it('uses raw SQL tile hydration to filter by the log body expression', async () => {
168+
const saved = await callTool(client, 'clickstack_save_dashboard', {
169+
name: 'Body fallback',
170+
tiles: [
171+
{
172+
name: 'Matching logs',
173+
config: {
174+
configType: 'sql',
175+
displayType: 'table',
176+
connectionId,
177+
sourceId: source._id.toString(),
178+
sqlTemplate:
179+
'SELECT Body FROM $__sourceTable WHERE $__timeFilter(Timestamp) AND $__filters',
180+
},
181+
},
182+
],
183+
});
184+
expect(saved.isError).toBeFalsy();
185+
const dashboard = JSON.parse(getFirstText(saved));
186+
const query = jest.spyOn(ClickhouseClient.prototype, 'queryChartConfig');
187+
try {
188+
const result = await callTool(client, 'clickstack_query_tile', {
189+
dashboardId: dashboard.id,
190+
tileId: dashboard.tiles[0].id,
191+
...currentWindow(),
192+
});
193+
expect(result.isError).toBeFalsy();
194+
expect(query).toHaveBeenCalledTimes(1);
195+
const [queryOptions] = query.mock.calls[0];
196+
if (!isRawSqlChartConfig(queryOptions.config)) {
197+
throw new Error('Expected a raw SQL tile query');
198+
}
199+
const clickhouseClient = new ClickhouseClient({
200+
host: config.CLICKHOUSE_HOST,
201+
username: config.CLICKHOUSE_USER,
202+
password: config.CLICKHOUSE_PASSWORD,
203+
});
204+
// query_tile exposes variables but not runtime dashboard filters.
205+
const filtered = await clickhouseClient.queryChartConfig({
206+
...queryOptions,
207+
config: {
208+
...queryOptions.config,
209+
filters: [{ type: 'lucene', condition: 'needle' }],
210+
},
211+
});
212+
expect(filtered.data).toEqual([{ Body: 'needle in body' }]);
213+
} finally {
214+
query.mockRestore();
215+
}
216+
});
217+
218+
it('uses the fallback for pattern samples, counts and drill-down filters', async () => {
219+
const result = await callTool(client, 'clickstack_event_patterns', {
220+
sourceId: source._id.toString(),
221+
where: 'needle',
222+
...currentWindow(),
223+
});
224+
expect(result.isError).toBeFalsy();
225+
const output = JSON.parse(getFirstText(result));
226+
expect(output.summary.totalCount).toBe(1);
227+
expect(output.summary.sampledCount).toBe(1);
228+
expect(output.patterns).toHaveLength(1);
229+
expect(output.patterns[0].whereSnippet).toContain('Body');
230+
const drilldown = await callTool(client, 'clickstack_search', {
231+
sourceId: source._id.toString(),
232+
where: output.patterns[0].whereSnippet,
233+
whereLanguage: 'sql',
234+
...currentWindow(),
235+
});
236+
expect(drilldown.isError).toBeFalsy();
237+
expect(JSON.parse(getFirstText(drilldown)).result.data).toEqual([
238+
expect.objectContaining({ Body: 'needle in body' }),
239+
]);
240+
});
241+
242+
it('uses the fallback in both emerging-signals windows', async () => {
243+
const result = await callTool(client, 'clickstack_emerging_signals', {
244+
sourceId: source._id.toString(),
245+
where: 'needle',
246+
currentStartTime: currentWindow().startTime,
247+
currentEndTime: currentWindow().endTime,
248+
baselineStartTime: new Date(now - 10 * 60_000).toISOString(),
249+
baselineEndTime: currentWindow().startTime,
250+
});
251+
expect(result.isError).toBeFalsy();
252+
const output = JSON.parse(getFirstText(result));
253+
expect(output.summary.currentWindow.total).toBe(1);
254+
expect(output.summary.currentWindow.sampled).toBe(1);
255+
expect(output.summary.baselineWindow.total).toBe(1);
256+
expect(output.summary.baselineWindow.sampled).toBe(1);
257+
});
258+
259+
it('uses the fallback in both event-deltas samples', async () => {
260+
const result = await callTool(client, 'clickstack_event_deltas', {
261+
sourceId: source._id.toString(),
262+
target: { ...currentWindow(), where: 'needle' },
263+
baseline: {
264+
startTime: new Date(now - 10 * 60_000).toISOString(),
265+
endTime: currentWindow().startTime,
266+
where: 'needle',
267+
},
268+
});
269+
expect(result.isError).toBeFalsy();
270+
const output = JSON.parse(getFirstText(result));
271+
expect(output.summary.targetSampleCount).toBe(1);
272+
expect(output.summary.baselineSampleCount).toBe(1);
273+
});
274+
});

‎packages/api/src/mcp/tools/query/eventDeltas.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,8 @@ export function registerEventDeltas({ context, registerTool }: ToolRegistrar) {
296296
'implicitColumnExpression' in source
297297
? source.implicitColumnExpression
298298
: undefined,
299+
bodyExpression:
300+
source.kind === SourceKind.Log ? source.bodyExpression : undefined,
299301
orderBy: stableSampleExpr,
300302
limit: { limit: sampleSize },
301303
dateRange: [startDate, endDate],

‎packages/api/src/mcp/tools/query/helpers.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -678,6 +678,8 @@ export async function runConfigTile(
678678
connection: source.connection.toString(),
679679
timestampValueExpression: source.timestampValueExpression,
680680
implicitColumnExpression: implicitColumn,
681+
bodyExpression:
682+
source.kind === SourceKind.Log ? source.bodyExpression : undefined,
681683
useTextIndexForImplicitColumn,
682684
dateRange: [startDate, endDate] as [Date, Date],
683685
variables: options?.variables,
@@ -738,6 +740,7 @@ export async function runConfigTile(
738740
let sourceFields: {
739741
from?: { databaseName: string; tableName: string };
740742
implicitColumnExpression?: string;
743+
bodyExpression?: string;
741744
useTextIndexForImplicitColumn?: UseTextIndex;
742745
metricTables?: MetricTable;
743746
} = {};
@@ -752,6 +755,8 @@ export async function runConfigTile(
752755
'implicitColumnExpression' in source
753756
? source.implicitColumnExpression
754757
: undefined,
758+
bodyExpression:
759+
source.kind === SourceKind.Log ? source.bodyExpression : undefined,
755760
useTextIndexForImplicitColumn:
756761
'useTextIndexForImplicitColumn' in source
757762
? source.useTextIndexForImplicitColumn

‎packages/api/src/mcp/tools/query/patternDrilldown.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ export async function userWhereAsSql({
8080
tableName,
8181
connectionId,
8282
implicitColumnExpression,
83+
bodyExpression,
8384
useTextIndexForImplicitColumn,
8485
}: {
8586
where: string;
@@ -90,6 +91,7 @@ export async function userWhereAsSql({
9091
tableName: string;
9192
connectionId: string;
9293
implicitColumnExpression?: string;
94+
bodyExpression?: string;
9395
useTextIndexForImplicitColumn?: UseTextIndex;
9496
}): Promise<string> {
9597
if (!where.trim()) return '';
@@ -103,6 +105,7 @@ export async function userWhereAsSql({
103105
from: { databaseName, tableName },
104106
connectionId,
105107
implicitColumnExpression,
108+
bodyExpression,
106109
useTextIndexForImplicitColumn,
107110
});
108111
}

‎packages/api/src/mcp/tools/query/runEventPatterns.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ import type {
1111
ChartConfigWithDateRange,
1212
ChartVariable,
1313
} from '@hyperdx/common-utils/dist/types';
14-
import { DisplayType } from '@hyperdx/common-utils/dist/types';
14+
import { DisplayType, SourceKind } from '@hyperdx/common-utils/dist/types';
1515
import { trace } from '@opentelemetry/api';
1616

1717
import { ClickhouseClient } from '@/clickhouse';
@@ -168,6 +168,8 @@ export async function mineWindowPatterns(
168168
'implicitColumnExpression' in source
169169
? source.implicitColumnExpression
170170
: undefined;
171+
const bodyExpression =
172+
source.kind === SourceKind.Log ? source.bodyExpression : undefined;
171173
const useTextIndexForImplicitColumn =
172174
'useTextIndexForImplicitColumn' in source
173175
? source.useTextIndexForImplicitColumn
@@ -190,6 +192,7 @@ export async function mineWindowPatterns(
190192
connection: source.connection.toString(),
191193
timestampValueExpression: source.timestampValueExpression,
192194
implicitColumnExpression: implicitColumn,
195+
bodyExpression,
193196
useTextIndexForImplicitColumn,
194197
orderBy: [{ ordering: 'DESC' as const, valueExpression: 'rand()' }],
195198
limit: { limit: sampleSize, offset: 0 },
@@ -211,6 +214,7 @@ export async function mineWindowPatterns(
211214
connection: source.connection.toString(),
212215
timestampValueExpression: source.timestampValueExpression,
213216
implicitColumnExpression: implicitColumn,
217+
bodyExpression,
214218
useTextIndexForImplicitColumn,
215219
limit: { limit: 1, offset: 0 },
216220
dateRange: [startDate, endDate] as [Date, Date],
@@ -232,6 +236,7 @@ export async function mineWindowPatterns(
232236
tableName: source.from.tableName,
233237
connectionId: source.connection.toString(),
234238
implicitColumnExpression: implicitColumn,
239+
bodyExpression,
235240
useTextIndexForImplicitColumn,
236241
}).catch(err => {
237242
if (queriesFailed || options.abortSignal?.aborted) return null;

0 commit comments

Comments
 (0)