Skip to content

fix(common-utils): support numeric range searches on String attributes - #3693

Open
Aryainguz wants to merge 10 commits into
ClickHouse:mainfrom
Aryainguz:aryainguz/numeric-attribute-comparisons
Open

Aryainguz wants to merge 10 commits into
ClickHouse:mainfrom
Aryainguz:aryainguz/numeric-attribute-comparisons

Conversation

@Aryainguz

@Aryainguz Aryainguz commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Attribute maps in the ClickStack schema store every value as a String, so Lucene comparisons on attributes compiled to string comparisons. LogAttributes.gc_pause_ms:>100 became LogAttributes['gc_pause_ms'] > '100', which matches '17.25' and '5' and misses '150'. Ranges with numeric bounds became ... BETWEEN 10 AND 20 on a String, which ClickHouse rejects ("no supertype for types String, UInt8").

When the field is String-typed and every bound is a numeric literal, gt, gte, lt, lte and range now compare toFloat64OrNull(field):

LogAttributes.gc_pause_ms:>100
before: (`LogAttributes`['gc_pause_ms'] > '100' AND indexHint(...))
after:  (ifNull(toFloat64OrNull(`LogAttributes`['gc_pause_ms']) > 100, 0) AND indexHint(...))
  • Values that aren't numbers convert to NULL and don't match. ifNull(..., 0) keeps that a plain non-match instead of NULL, so NOT field:>100 still includes rows where the attribute is missing or non-numeric, as it does today.
  • Unchanged: numeric columns (Duration:>100), JSON paths, and non-numeric bounds (version:>abc, hex such as 0x10), which still compare as strings.
  • It applies to any String-typed field, not only map subscripts, so an attribute behaves the same whether it is read from the map or from a materialized String column.
  • The indexHint(mapContains(...)) postfix is kept.

The existing tests mocked LogAttributes as a bare Map and only asserted SQL text, which is how BETWEEN 1 AND 5 on a String passed without ever running. One existing case, foo:[1 TO 5] on a String column, now expects the numeric form; its old expected SQL fails on ClickHouse.

Closes #3692

Screenshots or video

Same live demo data, Demo Logs, last 15 minutes. infra.gc_pause_ms peaks at 50 in this data.

Before (main) After (this branch)
gc_pause_ms:>100: no error, but 54,841 rows whose values (8.1, 16.13, 32…) are all below 100 0 results
image image
gc_pause_ms:[10 TO 20]: "no supertype for types String, UInt8" 11,668 results, all between 10 and 20 inclusive
image image

How to reproduce

Starting point: yarn app:dev:local, onboarding → Connect to Demo Server.

On main:

  1. Open Search, source Demo Logs, time range Last 15 minutes.
  2. Set SELECT to Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body.
  3. Search LogAttributes.infra.gc_pause_ms:>100: rows come back with values below 100.
  4. Search LogAttributes.infra.gc_pause_ms:[10 TO 20]: the query fails with the "no supertype" error.

On this branch:

  1. Same steps.
  2. Step 3 returns no rows (the demo data never exceeds 50).
  3. Step 4 returns rows whose gc_pause_ms is between 10 and 20.

Local validation

  • Ran the generated SQL on ClickHouse 26.8 against values 5, 10, 17.25, 20, 100, 150, -3, abc and a row without the key. Every form (>, >=, <, <=, negative bounds, [a TO b], {a TO b}, [a TO *], -field:[a TO b], NOT field:>x) returned exactly the expected rows. The old SQL returned 1,3,4,6,7 for >100 instead of 6.
  • queryParser.test.ts: 16 new cases (12 numeric, 4 unchanged).
  • packages/common-utils: 3,175 tests pass; ci:lint 0 errors, warning count unchanged (80).

How to test on Vercel preview

Preview routes: /search

Steps:

  1. Click Connect to Demo Server on the onboarding screen.
  2. Open /search with the Demo Logs source and Last 15 minutes.
  3. Set SELECT to Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body.
  4. Search LogAttributes.infra.gc_pause_ms:[10 TO 20].
  5. Verify results appear and every gc_pause_ms value is between 10 and 20.
  6. Search LogAttributes.infra.gc_pause_ms:>100 and confirm no results are returned.

References

… attributes

Attribute maps store every value as a String, so `field:>100` compared
text ('17.25' > '100' is true) and ranges with numeric bounds failed
with "no supertype for types String, UInt8". When every bound is a
numeric literal and the field is String-typed, compare
toFloat64OrNull(field) instead. ifNull keeps non-numeric and missing
values a non-match, so a surrounding NOT still includes them.
@Aryainguz
Aryainguz requested a review from a team as a code owner November 17, 2026 13:19
@vercel

vercel Bot commented Nov 17, 2026

Copy link
Copy Markdown

@Aryainguz is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: cfe3d1b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

1 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a tightly scoped, well-tested fix. The change is confined to SQLSerializer.range() plus three small helpers and a columnType plumbing field; >, <, >=, <= (gt/lt/gte/lte) are deliberately left lexical, which resolves the earlier reviewer concern that numeric coercion would silently break version:>2-style saved searches. The numeric path is correctly gated on ClickHouse String/FixedString column types (Enum/UUID/IP/numeric columns stay lexical), and ifNull(..., 0) preserves the existing negation semantics for missing/non-numeric values. Unit coverage plus a real-ClickHouse integration test (numericStringComparisons.int.test.ts, wired through jest.int.config.js/ci:int) exercise inclusive/exclusive/half-open/negated ranges and the materialized-column and FixedString cases.

A few low-severity items are worth noting but none block merge.

🔵 P3 nitpicks (2)
  • packages/common-utils/src/queryParser.ts:1811 — Attribute maps whose value type is wrapped (e.g. Map(LowCardinality(String), LowCardinality(String))) extract to 'Unknown' via the /,\s+(\w+)\)$/ value-type regex, so isStringComparedToNumbers returns false and a numeric-bound range still emits BETWEEN on a String, continuing to fail with "no supertype" on those non-default schemas; the fix only covers unwrapped String/FixedString.
    • Fix: Apply stripTypeWrappers to the extracted map value type before classifying it, or document this as a known limitation in the changeset.
  • packages/common-utils/src/__tests__/numericStringComparisons.int.test.ts:125 — The upper-bounded/open-start numeric form [* TO N] on a String attribute is not directly asserted; only [N TO *], [N TO M], {N TO M}, and the negated form are covered, leaving the end-only branch of compareBound unverified.
    • Fix: Add one LogAttributes.gc_pause_ms:[* TO 100] case to the integration it.each table.

Reviewers (8): correctness, adversarial, security, testing, maintainability, kieran-typescript, previous-comments, project-standards.

Testing gaps:

  • [* TO N] numeric range on a String attribute is untested (the symmetric half-open case).
  • Numeric ranges on genuinely textual String columns (e.g. ServiceName:[1 TO 5]) now reinterpret numerically instead of erroring; not a regression (the prior SQL errored), but no assertion pins this intended behavior for a non-attribute text column.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@Aryainguz
Aryainguz marked this pull request as draft November 17, 2026 13:33
…istent

- `-field:>x` is rewritten to the opposite operator, so missing and
  non-numeric values now match it, as for `-field:[a TO b]` and NOT.
- Only String and FixedString columns get toFloat64OrNull. Enum, UUID
  and IP types also map to JSDataType.String but ClickHouse rejects
  toFloat64OrNull for them, and Enum already compares to numbers.
- Integration test runs each search against ClickHouse and checks the
  returned rows.
@Aryainguz
Aryainguz marked this pull request as ready for review November 17, 2026 13:35
@pulpdrew

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

However, I have a concern: Today <, >, etc. compare String fields lexically, which works for fixed-width values (ISO dates, zero-padded codes) and roughly for version strings. This change switches to numeric comparison whenever the bound looks like a number, and values that don't parse as numbers silently stop matching. For example, version:>2 used to match 2.0.2 and now matches nothing, because 2.0.2 isn't a valid number. Saved searches and alerts relying on this would return empty results without any error

I think as an alternative proposal would look like this:

  1. Keep the range syntax fix - those are already explicitly numeric, so we're only changing things that errored before. Then, users can use that syntax when they need numeric comparison against a string value.
  2. <, > already work against values that ClickHouse stores as numeric (top level columns, or Map(String, Float)). So there are a number of workarounds available that don't break compatibility: use the range syntax from (1), promote a field to a top-level numeric column, or use SQL to filter and apply the desired cast there.

`>`, `<`, `>=` and `<=` on String fields compare as text again. Fixed-width
values such as dates and versions rely on that, and switching to a numeric
comparison would make those searches match nothing without an error.

Ranges with numeric bounds still compare with toFloat64OrNull, since they
failed with "no supertype" before.
@Aryainguz Aryainguz changed the title fix(common-utils): compare numeric search bounds as numbers on String attributes fix(common-utils): support numeric range searches on String attributes Nov 17, 2026
@Aryainguz

Aryainguz commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Sure @pulpdrew, since this PR was about ranges, So I've cut it down to just that.

>, <, >= and <= on text fields work the same as on main again, so saved searches and alerts don't change.
Ranges with numbers, like field:[10 TO 20], now compare as numbers. Before, they failed with an error.

One case still fails, maps whose values are wrapped types, like Map(String, LowCardinality(String)), still give the old error on ranges. I left it out because the fix is in the code that reads map value types, which every map search uses, so I'd rather change that in its own PR than add it here. The default ClickStack schema stores map values as plain String, so ranges work there.

Thanks for catching this!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Numeric searches on attributes compare text: field:>100 matches 17.25, and field:[10 TO 20] errors

2 participants