Repository navigation
fix(common-utils): honor open, exclusive and non-numeric range bounds - #2779
Conversation
SQLSerializer.range() serialized every Lucene range as an inclusive
BETWEEN over parseFloat'd bounds, so three documented forms compiled to
SQL that ClickHouse either rejects or silently answers wrong:
Duration:[* TO 500] -> Duration BETWEEN '*' AND 500
Code 53 TYPE_MISMATCH, "Cannot convert string '*' to type UInt64".
Duration:{100 TO 500} -> Duration BETWEEN 100 AND 500
The ranged-term node's `inclusive` flag was never read, so {} and
both half-open forms came out inclusive.
Timestamp:[2024-01-01 TO 2024-06-01] -> Timestamp BETWEEN 2024 AND 2024
Number.parseFloat stops at the first '-', so both bounds collapsed
to 2024 and the query matched nothing.
range() now reads `inclusive` from the node and picks > / >= and < / <=
from it, drops a bound entirely when it is the unbounded '*', and falls
back to isNotNull() when both bounds are open -- the same expression
`field:*` already emits. attemptToParseNumber switches from
Number.parseFloat to Number, so a bound is converted only when the whole
string is numeric; a date or any other partially-numeric bound reaches
ClickHouse as a quoted string, which it coerces to the column type.
A closed inclusive range still emits BETWEEN, and every bound parseFloat
consumed in full converts to the same number it did before -- including
Infinity, -Infinity and 1e400, which ClickHouse accepts. Only the bounds
parseFloat was truncating change. Compared over 7,251 generated bound
strings: nothing differs except where parseFloat was reading a prefix.
EnglishSerializer.range takes the same flag, so `Duration:{100 TO 500}`
now reads "between 100 (exclusive) and 500 (exclusive)" rather than
describing an inclusive range while the SQL excludes both ends.
Checked against ClickHouse 26.7.1 on a DateTime64/UInt64 table: the
emitted `Duration <= 500`, `Duration > 100 AND Duration < 500` and
`Timestamp BETWEEN '2024-01-01' AND '2024-06-01'` return the expected
rows where the previous output errored or returned none.
|
@shuvamk is attempting to deploy a commit to the HyperDX Team on Vercel. A member of the Team first needs to authorize it. |
🦋 Changeset detectedLatest commit: 832c351 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Greptile SummaryThe PR updates the shared Lucene query serializer to preserve open, exclusive, and non-numeric range-bound semantics.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| packages/common-utils/src/queryParser.ts | Updates SQL and English range serialization to honor unbounded and exclusive endpoints while preserving non-numeric bounds. |
| packages/common-utils/src/tests/queryParser.test.ts | Adds coverage for inclusive, exclusive, half-open, unbounded, negated, map-key, and date range serialization. |
| .changeset/range-open-exclusive-bounds.md | Documents the corrected range behavior and publishes it as a common-utils patch. |
Reviews (2): Last reviewed commit: "Merge branch 'main' into claude/fix-rang..." | Re-trigger Greptile
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
pulpdrew
left a comment
There was a problem hiding this comment.
LGTM, this is a nice improvement
## Summary This PR adds new in-app lucene syntax examples for the new lucene support added in #2779 ### Screenshots or video <img width="738" height="464" alt="Screenshot 2026-08-07 at 3 29 07 PM" src="https://github.com/user-attachments/assets/626ea430-c105-4a62-8743-7776367b59bf" /> ### How to test on Vercel preview This can be viewed in the preview environment by clicking on the (?) on the SQL/Lucene language switcher. ### References - Linear Issue: - Related PRs:
#3693) ## Summary Attribute maps in the ClickStack schema store every value as a `String`, so Lucene comparisons on attributes compiled to string comparisons. `LogAttributes.gc_pause_ms:>100` became `LogAttributes['gc_pause_ms'] > '100'`, which matches `'17.25'` and `'5'` and misses `'150'`. Ranges with numeric bounds became `... BETWEEN 10 AND 20` on a String, which ClickHouse rejects ("no supertype for types String, UInt8"). When the field is String-typed and every bound is a numeric literal, `gt`, `gte`, `lt`, `lte` and `range` now compare `toFloat64OrNull(field)`: ``` LogAttributes.gc_pause_ms:>100 before: (`LogAttributes`['gc_pause_ms'] > '100' AND indexHint(...)) after: (ifNull(toFloat64OrNull(`LogAttributes`['gc_pause_ms']) > 100, 0) AND indexHint(...)) ``` - Values that aren't numbers convert to NULL and don't match. `ifNull(..., 0)` keeps that a plain non-match instead of NULL, so `NOT field:>100` still includes rows where the attribute is missing or non-numeric, as it does today. - Unchanged: numeric columns (`Duration:>100`), JSON paths, and non-numeric bounds (`version:>abc`, hex such as `0x10`), which still compare as strings. - It applies to any String-typed field, not only map subscripts, so an attribute behaves the same whether it is read from the map or from a materialized String column. - The `indexHint(mapContains(...))` postfix is kept. The existing tests mocked `LogAttributes` as a bare `Map` and only asserted SQL text, which is how `BETWEEN 1 AND 5` on a String passed without ever running. One existing case, `foo:[1 TO 5]` on a String column, now expects the numeric form; its old expected SQL fails on ClickHouse. Closes #3692 ### Screenshots or video Same live demo data, Demo Logs, last 15 minutes. `infra.gc_pause_ms` peaks at 50 in this data. | Before (`main`) | After (this branch) | | :----- | :---- | | `gc_pause_ms:>100`: no error, but 54,841 rows whose values (8.1, 16.13, 32…) are all below 100 | 0 results | | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/05018a18-3add-4935-ac09-b4ac7f9f5607" /> | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/28e6de55-dcc0-45c4-880f-308c7af1ccf5" /> | | `gc_pause_ms:[10 TO 20]`: "no supertype for types String, UInt8" | 11,668 results, all between 10 and 20 inclusive | | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/d942a870-ddd7-410d-a2e7-342c28944424" /> | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/310bb202-f043-4a54-9675-3611cb029f97" /> | ### How to reproduce **Starting point:** `yarn app:dev:local`, onboarding → **Connect to Demo Server**. **On `main`:** 1. Open **Search**, source **Demo Logs**, time range **Last 15 minutes**. 2. Set SELECT to `Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body`. 3. Search `LogAttributes.infra.gc_pause_ms:>100`: rows come back with values below 100. 4. Search `LogAttributes.infra.gc_pause_ms:[10 TO 20]`: the query fails with the "no supertype" error. **On this branch:** 1. Same steps. 2. Step 3 returns no rows (the demo data never exceeds 50). 3. Step 4 returns rows whose `gc_pause_ms` is between 10 and 20. ### Local validation - Ran the generated SQL on ClickHouse 26.8 against values `5, 10, 17.25, 20, 100, 150, -3, abc` and a row without the key. Every form (`>`, `>=`, `<`, `<=`, negative bounds, `[a TO b]`, `{a TO b}`, `[a TO *]`, `-field:[a TO b]`, `NOT field:>x`) returned exactly the expected rows. The old SQL returned `1,3,4,6,7` for `>100` instead of `6`. - `queryParser.test.ts`: 16 new cases (12 numeric, 4 unchanged). - `packages/common-utils`: 3,175 tests pass; `ci:lint` 0 errors, warning count unchanged (80). ### How to test on Vercel preview **Preview routes:** /search **Steps:** 1. Click **Connect to Demo Server** on the onboarding screen. 2. Open /search with the **Demo Logs** source and **Last 15 minutes**. 3. Set SELECT to `Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body`. 4. Search `LogAttributes.infra.gc_pause_ms:[10 TO 20]`. 5. Verify results appear and every `gc_pause_ms` value is between 10 and 20. 6. Search `LogAttributes.infra.gc_pause_ms:>100` and confirm no results are returned. ### References - Related PRs: #2779 (range fixes, verified on numeric columns; its map-attribute range test asserted SQL that fails on a String map), #2841 ### Update: After review, this PR now only fixes range searches like field:[10 TO 20] on text fields. >, <, >= and <= work the same as on main (they compare as text) so existing saved searches and alerts don't change. Use a range like field:[100 TO *] for a numeric comparison. The >100 parts below no longer apply.
Summary
SQLSerializer.range()serializes every Lucene range as an inclusiveBETWEENoverparseFloat'd bounds. Three documented range forms compile to SQL that ClickHouse rejects or silently answers wrong (checked on ClickHouse 26.7.1):Duration:[* TO 500]Duration BETWEEN '*' AND 500TYPE_MISMATCH: Cannot convert string '*' to type UInt64Duration:{100 TO 500}Duration BETWEEN 100 AND 500Timestamp:[2024-01-01 TO 2024-06-01]Timestamp BETWEEN 2024 AND 2024Three causes:
*is never recognised as an unbounded side; the ranged-term node'sinclusiveflag is never read, so{},[}and{]all come out inclusive; andNumber.parseFloat('2024-01-01')stops at the first-.range()now takesinclusivefrom the node and picks>/>=and</<=from it, drops a bound that is*, and falls back toisNotNull()when both sides are open — the same expressionfield:*already emits.attemptToParseNumberusesNumber(), so a bound converts only when the whole string is numeric; a date bound reaches ClickHouse as a quoted string and is coerced to the column type. A closed inclusive range still emitsBETWEEN, and every boundparseFloatconsumed in full —Infinityand1e400included — converts to the same number it did before, so only the bounds it was truncating change.EnglishSerializer.rangetakes the same flag, so the search bar no longer describes an exclusive range as inclusive.Tests: 17 cases in
queryParser.test.ts— SQL for open, exclusive, half-open, negated, map-key and date ranges, plus the English explanation. 15 fail without the source change and pass with it; two pin the unchanged[100 TO 500]output.packages/common-utils: 1604 tests passing,ci:lint0 errors and the warning count unchanged. (Twopackages/apphook suites fail on my machine identically with and without this diff.)The in-app syntax reference documents ranges as numeric, so if you'd rather they stay numeric-only, say the word and I'll drop the
parseFloathalf — the*and{}halves stand on their own.Screenshots or video
N/A — non-UI change.
How to test on Vercel preview
N/A — non-UI change (
packages/common-utilsquery compiler).References