Skip to content

fix(common-utils): honor open, exclusive and non-numeric range bounds - #2779

Merged
kodiakhq[bot] merged 2 commits into
ClickHouse:mainfrom
shuvamk:claude/fix-range-bounds
Aug 5, 2026
Merged

kodiakhq[bot] merged 2 commits into
ClickHouse:mainfrom
shuvamk:claude/fix-range-bounds

Conversation

@shuvamk

@shuvamk shuvamk commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

SQLSerializer.range() serializes every Lucene range as an inclusive BETWEEN over parseFloat'd bounds. Three documented range forms compile to SQL that ClickHouse rejects or silently answers wrong (checked on ClickHouse 26.7.1):

Search Compiles to Result
Duration:[* TO 500] Duration BETWEEN '*' AND 500 Code 53 TYPE_MISMATCH: Cannot convert string '*' to type UInt64
Duration:{100 TO 500} Duration BETWEEN 100 AND 500 returns 100 and 500, which the user excluded
Timestamp:[2024-01-01 TO 2024-06-01] Timestamp BETWEEN 2024 AND 2024 0 rows

Three causes: * is never recognised as an unbounded side; the ranged-term node's inclusive flag is never read, so {}, [} and {] all come out inclusive; and Number.parseFloat('2024-01-01') stops at the first -.

range() now takes inclusive from the node and picks >/>= and </<= from it, drops a bound that is *, and falls back to isNotNull() when both sides are open — the same expression field:* already emits. attemptToParseNumber uses Number(), so a bound converts only when the whole string is numeric; a date bound reaches ClickHouse as a quoted string and is coerced to the column type. A closed inclusive range still emits BETWEEN, and every bound parseFloat consumed in full — Infinity and 1e400 included — converts to the same number it did before, so only the bounds it was truncating change. EnglishSerializer.range takes the same flag, so the search bar no longer describes an exclusive range as inclusive.

Tests: 17 cases in queryParser.test.ts — SQL for open, exclusive, half-open, negated, map-key and date ranges, plus the English explanation. 15 fail without the source change and pass with it; two pin the unchanged [100 TO 500] output. packages/common-utils: 1604 tests passing, ci:lint 0 errors and the warning count unchanged. (Two packages/app hook suites fail on my machine identically with and without this diff.)

The in-app syntax reference documents ranges as numeric, so if you'd rather they stay numeric-only, say the word and I'll drop the parseFloat half — the * and {} halves stand on their own.

Screenshots or video

N/A — non-UI change.

How to test on Vercel preview

N/A — non-UI change (packages/common-utils query compiler).

References

  • Linear Issue: N/A — external contribution, no Linear ticket
  • Related PRs: none

SQLSerializer.range() serialized every Lucene range as an inclusive
BETWEEN over parseFloat'd bounds, so three documented forms compiled to
SQL that ClickHouse either rejects or silently answers wrong:

  Duration:[* TO 500]                  -> Duration BETWEEN '*' AND 500
      Code 53 TYPE_MISMATCH, "Cannot convert string '*' to type UInt64".
  Duration:{100 TO 500}                -> Duration BETWEEN 100 AND 500
      The ranged-term node's `inclusive` flag was never read, so {} and
      both half-open forms came out inclusive.
  Timestamp:[2024-01-01 TO 2024-06-01] -> Timestamp BETWEEN 2024 AND 2024
      Number.parseFloat stops at the first '-', so both bounds collapsed
      to 2024 and the query matched nothing.

range() now reads `inclusive` from the node and picks > / >= and < / <=
from it, drops a bound entirely when it is the unbounded '*', and falls
back to isNotNull() when both bounds are open -- the same expression
`field:*` already emits. attemptToParseNumber switches from
Number.parseFloat to Number, so a bound is converted only when the whole
string is numeric; a date or any other partially-numeric bound reaches
ClickHouse as a quoted string, which it coerces to the column type.

A closed inclusive range still emits BETWEEN, and every bound parseFloat
consumed in full converts to the same number it did before -- including
Infinity, -Infinity and 1e400, which ClickHouse accepts. Only the bounds
parseFloat was truncating change. Compared over 7,251 generated bound
strings: nothing differs except where parseFloat was reading a prefix.

EnglishSerializer.range takes the same flag, so `Duration:{100 TO 500}`
now reads "between 100 (exclusive) and 500 (exclusive)" rather than
describing an inclusive range while the SQL excludes both ends.

Checked against ClickHouse 26.7.1 on a DateTime64/UInt64 table: the
emitted `Duration <= 500`, `Duration > 100 AND Duration < 500` and
`Timestamp BETWEEN '2024-01-01' AND '2024-06-01'` return the expected
rows where the previous output errored or returned none.
@vercel

vercel Bot commented Aug 3, 2026

Copy link
Copy Markdown

@shuvamk is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@changeset-bot

changeset-bot Bot commented Aug 3, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 832c351

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@hyperdx/common-utils Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@greptile-apps

greptile-apps Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR updates the shared Lucene query serializer to preserve open, exclusive, and non-numeric range-bound semantics.

  • Emits one-sided comparisons for unbounded ranges and bound-specific operators for exclusive or half-open ranges.
  • Uses whole-string numeric conversion so date-like bounds remain quoted strings.
  • Passes range inclusivity into English explanations and adds focused SQL and explanation tests.
  • Adds a patch changeset for @hyperdx/common-utils.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
packages/common-utils/src/queryParser.ts Updates SQL and English range serialization to honor unbounded and exclusive endpoints while preserving non-numeric bounds.
packages/common-utils/src/tests/queryParser.test.ts Adds coverage for inclusive, exclusive, half-open, unbounded, negated, map-key, and date range serialization.
.changeset/range-open-exclusive-bounds.md Documents the corrected range behavior and publishes it as a common-utils patch.

Reviews (2): Last reviewed commit: "Merge branch 'main' into claude/fix-rang..." | Re-trigger Greptile

@pulpdrew
pulpdrew self-requested a review August 4, 2026 19:58
@vercel

vercel Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hyperdx-oss Ready Ready Preview Aug 5, 2026 1:35pm

Request Review

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, this is a nice improvement

@kodiakhq
kodiakhq Bot merged commit 6a35df0 into ClickHouse:main Aug 5, 2026
23 of 26 checks passed
kodiakhq Bot pushed a commit that referenced this pull request Aug 7, 2026
## Summary

This PR adds new in-app lucene syntax examples for the new lucene support added in #2779

### Screenshots or video

<img width="738" height="464" alt="Screenshot 2026-08-07 at 3 29 07 PM" src="https://github.com/user-attachments/assets/626ea430-c105-4a62-8743-7776367b59bf" />

### How to test on Vercel preview

This can be viewed in the preview environment by clicking on the (?) on the SQL/Lucene language switcher.

### References



- Linear Issue:
- Related PRs:
@shuvamk
shuvamk deleted the claude/fix-range-bounds branch August 8, 2026 09:45
kodiakhq Bot pushed a commit that referenced this pull request Nov 18, 2026
#3693)

## Summary

Attribute maps in the ClickStack schema store every value as a `String`, so Lucene comparisons on attributes compiled to string comparisons. `LogAttributes.gc_pause_ms:>100` became `LogAttributes['gc_pause_ms'] > '100'`, which matches `'17.25'` and `'5'` and misses `'150'`. Ranges with numeric bounds became `... BETWEEN 10 AND 20` on a String, which ClickHouse rejects ("no supertype for types String, UInt8").

When the field is String-typed and every bound is a numeric literal, `gt`, `gte`, `lt`, `lte` and `range` now compare `toFloat64OrNull(field)`:

```
LogAttributes.gc_pause_ms:>100
before: (`LogAttributes`['gc_pause_ms'] > '100' AND indexHint(...))
after:  (ifNull(toFloat64OrNull(`LogAttributes`['gc_pause_ms']) > 100, 0) AND indexHint(...))
```

- Values that aren't numbers convert to NULL and don't match. `ifNull(..., 0)` keeps that a plain non-match instead of NULL, so `NOT field:>100` still includes rows where the attribute is missing or non-numeric, as it does today.
- Unchanged: numeric columns (`Duration:>100`), JSON paths, and non-numeric bounds (`version:>abc`, hex such as `0x10`), which still compare as strings.
- It applies to any String-typed field, not only map subscripts, so an attribute behaves the same whether it is read from the map or from a materialized String column.
- The `indexHint(mapContains(...))` postfix is kept.

The existing tests mocked `LogAttributes` as a bare `Map` and only asserted SQL text, which is how `BETWEEN 1 AND 5` on a String passed without ever running. One existing case, `foo:[1 TO 5]` on a String column, now expects the numeric form; its old expected SQL fails on ClickHouse.

Closes #3692

### Screenshots or video

Same live demo data, Demo Logs, last 15 minutes. `infra.gc_pause_ms` peaks at 50 in this data.

| Before (`main`) | After (this branch) |
| :----- | :---- |
| `gc_pause_ms:>100`: no error, but 54,841 rows whose values (8.1, 16.13, 32…) are all below 100 | 0 results |
| <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/05018a18-3add-4935-ac09-b4ac7f9f5607" /> | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/28e6de55-dcc0-45c4-880f-308c7af1ccf5" /> |
| `gc_pause_ms:[10 TO 20]`: "no supertype for types String, UInt8" | 11,668 results, all between 10 and 20 inclusive |
| <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/d942a870-ddd7-410d-a2e7-342c28944424" /> | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/310bb202-f043-4a54-9675-3611cb029f97" /> |

### How to reproduce

**Starting point:** `yarn app:dev:local`, onboarding → **Connect to Demo Server**.

**On `main`:**

1. Open **Search**, source **Demo Logs**, time range **Last 15 minutes**.
2. Set SELECT to `Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body`.
3. Search `LogAttributes.infra.gc_pause_ms:>100`: rows come back with values below 100.
4. Search `LogAttributes.infra.gc_pause_ms:[10 TO 20]`: the query fails with the "no supertype" error.

**On this branch:**

1. Same steps.
2. Step 3 returns no rows (the demo data never exceeds 50).
3. Step 4 returns rows whose `gc_pause_ms` is between 10 and 20.

### Local validation

- Ran the generated SQL on ClickHouse 26.8 against values `5, 10, 17.25, 20, 100, 150, -3, abc` and a row without the key. Every form (`>`, `>=`, `<`, `<=`, negative bounds, `[a TO b]`, `{a TO b}`, `[a TO *]`, `-field:[a TO b]`, `NOT field:>x`) returned exactly the expected rows. The old SQL returned `1,3,4,6,7` for `>100` instead of `6`.
- `queryParser.test.ts`: 16 new cases (12 numeric, 4 unchanged).
- `packages/common-utils`: 3,175 tests pass; `ci:lint` 0 errors, warning count unchanged (80).

### How to test on Vercel preview

**Preview routes:** /search

**Steps:**

1. Click **Connect to Demo Server** on the onboarding screen.
2. Open /search with the **Demo Logs** source and **Last 15 minutes**.
3. Set SELECT to `Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body`.
4. Search `LogAttributes.infra.gc_pause_ms:[10 TO 20]`.
5. Verify results appear and every `gc_pause_ms` value is between 10 and 20.
6. Search `LogAttributes.infra.gc_pause_ms:>100` and confirm no results are returned.

### References

- Related PRs: #2779 (range fixes, verified on numeric columns; its map-attribute range test asserted SQL that fails on a String map), #2841

### Update: 
After review, this PR now only fixes range searches like field:[10 TO 20] on text fields. >, <, >= and <= work the same as on main (they compare as text) so existing saved searches and alerts don't change. Use a range like field:[100 TO *] for a numeric comparison. The >100 parts below no longer apply.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants