Repository navigation
Conversation
… attributes
Attribute maps store every value as a String, so `field:>100` compared
text ('17.25' > '100' is true) and ranges with numeric bounds failed
with "no supertype for types String, UInt8". When every bound is a
numeric literal and the field is String-typed, compare
toFloat64OrNull(field) instead. ifNull keeps non-numeric and missing
values a non-match, so a surrounding NOT still includes them.
|
@Aryainguz is attempting to deploy a commit to the HyperDX Team on Vercel. A member of the Team first needs to authorize it. |
🦋 Changeset detectedLatest commit: cfe3d1b The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
PR Review
1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor 1 posted as inline comment(s) on the changed lines. Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
Deep Review✅ No critical issues found. This is a tightly scoped, well-tested fix. The change is confined to A few low-severity items are worth noting but none block merge. 🔵 P3 nitpicks (2)
Reviewers (8): correctness, adversarial, security, testing, maintainability, kieran-typescript, previous-comments, project-standards. Testing gaps:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
…istent - `-field:>x` is rewritten to the opposite operator, so missing and non-numeric values now match it, as for `-field:[a TO b]` and NOT. - Only String and FixedString columns get toFloat64OrNull. Enum, UUID and IP types also map to JSDataType.String but ClickHouse rejects toFloat64OrNull for them, and Enum already compares to numbers. - Integration test runs each search against ClickHouse and checks the returned rows.
|
Thanks for the contribution! However, I have a concern: Today <, >, etc. compare String fields lexically, which works for fixed-width values (ISO dates, zero-padded codes) and roughly for version strings. This change switches to numeric comparison whenever the bound looks like a number, and values that don't parse as numbers silently stop matching. For example, version:>2 used to match 2.0.2 and now matches nothing, because 2.0.2 isn't a valid number. Saved searches and alerts relying on this would return empty results without any error I think as an alternative proposal would look like this:
|
`>`, `<`, `>=` and `<=` on String fields compare as text again. Fixed-width values such as dates and versions rely on that, and switching to a numeric comparison would make those searches match nothing without an error. Ranges with numeric bounds still compare with toFloat64OrNull, since they failed with "no supertype" before.
|
Sure @pulpdrew, since this PR was about ranges, So I've cut it down to just that.
One case still fails, maps whose values are wrapped types, like Map(String, LowCardinality(String)), still give the old error on ranges. I left it out because the fix is in the code that reads map value types, which every map search uses, so I'd rather change that in its own PR than add it here. The default ClickStack schema stores map values as plain String, so ranges work there. Thanks for catching this! |
…arisons' into aryainguz/numeric-attribute-comparisons
Summary
Attribute maps in the ClickStack schema store every value as a
String, so Lucene comparisons on attributes compiled to string comparisons.LogAttributes.gc_pause_ms:>100becameLogAttributes['gc_pause_ms'] > '100', which matches'17.25'and'5'and misses'150'. Ranges with numeric bounds became... BETWEEN 10 AND 20on a String, which ClickHouse rejects ("no supertype for types String, UInt8").When the field is String-typed and every bound is a numeric literal,
gt,gte,lt,lteandrangenow comparetoFloat64OrNull(field):ifNull(..., 0)keeps that a plain non-match instead of NULL, soNOT field:>100still includes rows where the attribute is missing or non-numeric, as it does today.Duration:>100), JSON paths, and non-numeric bounds (version:>abc, hex such as0x10), which still compare as strings.indexHint(mapContains(...))postfix is kept.The existing tests mocked
LogAttributesas a bareMapand only asserted SQL text, which is howBETWEEN 1 AND 5on a String passed without ever running. One existing case,foo:[1 TO 5]on a String column, now expects the numeric form; its old expected SQL fails on ClickHouse.Closes #3692
Screenshots or video
Same live demo data, Demo Logs, last 15 minutes.
infra.gc_pause_mspeaks at 50 in this data.main)gc_pause_ms:>100: no error, but 54,841 rows whose values (8.1, 16.13, 32…) are all below 100gc_pause_ms:[10 TO 20]: "no supertype for types String, UInt8"How to reproduce
Starting point:
yarn app:dev:local, onboarding → Connect to Demo Server.On
main:Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body.LogAttributes.infra.gc_pause_ms:>100: rows come back with values below 100.LogAttributes.infra.gc_pause_ms:[10 TO 20]: the query fails with the "no supertype" error.On this branch:
gc_pause_msis between 10 and 20.Local validation
5, 10, 17.25, 20, 100, 150, -3, abcand a row without the key. Every form (>,>=,<,<=, negative bounds,[a TO b],{a TO b},[a TO *],-field:[a TO b],NOT field:>x) returned exactly the expected rows. The old SQL returned1,3,4,6,7for>100instead of6.queryParser.test.ts: 16 new cases (12 numeric, 4 unchanged).packages/common-utils: 3,175 tests pass;ci:lint0 errors, warning count unchanged (80).How to test on Vercel preview
Preview routes: /search
Steps:
Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body.LogAttributes.infra.gc_pause_ms:[10 TO 20].gc_pause_msvalue is between 10 and 20.LogAttributes.infra.gc_pause_ms:>100and confirm no results are returned.References