Repository navigation
fix(common-utils): escape Map subscripts once in numeric and Bool searches - #2841
Merged
kodiakhq[bot] merged 3 commits intoAug 10, 2026
Merged
Conversation
…rches
Searching a Map column whose value type is numeric or Bool wrapped the map
subscript in a second layer of backticks, so ClickHouse read it as a single
identifier rather than a map lookup:
Measures.latency_ms:250 -> ```Measures``['latency_ms']` = CAST('250', 'Float64')
Flags.cached:true -> ```Flags``['cached']` = 1
getColumnForField() returns `column` already rendered for a map key
(`Measures`['latency_ms']) and bare for a plain column. Three equality
branches passed it as a SqlString `??` placeholder, which applies escapeId to
it again; every sibling branch (eq/Number, gte, lte, lt, gt, range, ILIKE)
interpolates it raw. For Map(String, Float64) only the unquoted spelling was
affected, since the quoted form routes through eq/Number; for
Map(String, Bool) both spellings were.
Interpolating raw at all three sites would have dropped the backticks a plain
Bool or numeric column gets today - `IsError` = 1 would become IsError = 1 -
so the escaping is kept and skipped only when the expression is already a
rendered Map subscript, which getColumnForField now reports via mapKey.
Fixes ClickHouse#2765
🦋 Changeset detectedLatest commit: 192c766 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@shuvamk is attempting to deploy a commit to the HyperDX Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Greptile SummaryThe PR corrects numeric and Bool Map equality serialization by preserving already-rendered Map subscripts while retaining identifier escaping for ordinary and materialized columns.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| packages/common-utils/src/queryParser.ts | Carries Map-key metadata into equality serialization and avoids applying identifier escaping twice to pre-rendered Map subscripts. |
| packages/common-utils/src/tests/queryParser.test.ts | Adds focused SQL-output regressions for numeric and Bool Maps while pinning plain and materialized column escaping. |
| .changeset/map-subscript-double-escape.md | Records the query-serialization fix as a patch release for the affected package. |
Reviews (3): Last reviewed commit: "Merge branch 'main' into claude/escape-m..." | Re-trigger Greptile
pulpdrew
approved these changes
Aug 10, 2026
kodiakhq Bot
pushed a commit
that referenced
this pull request
Nov 18, 2026
#3693) ## Summary Attribute maps in the ClickStack schema store every value as a `String`, so Lucene comparisons on attributes compiled to string comparisons. `LogAttributes.gc_pause_ms:>100` became `LogAttributes['gc_pause_ms'] > '100'`, which matches `'17.25'` and `'5'` and misses `'150'`. Ranges with numeric bounds became `... BETWEEN 10 AND 20` on a String, which ClickHouse rejects ("no supertype for types String, UInt8"). When the field is String-typed and every bound is a numeric literal, `gt`, `gte`, `lt`, `lte` and `range` now compare `toFloat64OrNull(field)`: ``` LogAttributes.gc_pause_ms:>100 before: (`LogAttributes`['gc_pause_ms'] > '100' AND indexHint(...)) after: (ifNull(toFloat64OrNull(`LogAttributes`['gc_pause_ms']) > 100, 0) AND indexHint(...)) ``` - Values that aren't numbers convert to NULL and don't match. `ifNull(..., 0)` keeps that a plain non-match instead of NULL, so `NOT field:>100` still includes rows where the attribute is missing or non-numeric, as it does today. - Unchanged: numeric columns (`Duration:>100`), JSON paths, and non-numeric bounds (`version:>abc`, hex such as `0x10`), which still compare as strings. - It applies to any String-typed field, not only map subscripts, so an attribute behaves the same whether it is read from the map or from a materialized String column. - The `indexHint(mapContains(...))` postfix is kept. The existing tests mocked `LogAttributes` as a bare `Map` and only asserted SQL text, which is how `BETWEEN 1 AND 5` on a String passed without ever running. One existing case, `foo:[1 TO 5]` on a String column, now expects the numeric form; its old expected SQL fails on ClickHouse. Closes #3692 ### Screenshots or video Same live demo data, Demo Logs, last 15 minutes. `infra.gc_pause_ms` peaks at 50 in this data. | Before (`main`) | After (this branch) | | :----- | :---- | | `gc_pause_ms:>100`: no error, but 54,841 rows whose values (8.1, 16.13, 32…) are all below 100 | 0 results | | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/05018a18-3add-4935-ac09-b4ac7f9f5607" /> | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/28e6de55-dcc0-45c4-880f-308c7af1ccf5" /> | | `gc_pause_ms:[10 TO 20]`: "no supertype for types String, UInt8" | 11,668 results, all between 10 and 20 inclusive | | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/d942a870-ddd7-410d-a2e7-342c28944424" /> | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/310bb202-f043-4a54-9675-3611cb029f97" /> | ### How to reproduce **Starting point:** `yarn app:dev:local`, onboarding → **Connect to Demo Server**. **On `main`:** 1. Open **Search**, source **Demo Logs**, time range **Last 15 minutes**. 2. Set SELECT to `Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body`. 3. Search `LogAttributes.infra.gc_pause_ms:>100`: rows come back with values below 100. 4. Search `LogAttributes.infra.gc_pause_ms:[10 TO 20]`: the query fails with the "no supertype" error. **On this branch:** 1. Same steps. 2. Step 3 returns no rows (the demo data never exceeds 50). 3. Step 4 returns rows whose `gc_pause_ms` is between 10 and 20. ### Local validation - Ran the generated SQL on ClickHouse 26.8 against values `5, 10, 17.25, 20, 100, 150, -3, abc` and a row without the key. Every form (`>`, `>=`, `<`, `<=`, negative bounds, `[a TO b]`, `{a TO b}`, `[a TO *]`, `-field:[a TO b]`, `NOT field:>x`) returned exactly the expected rows. The old SQL returned `1,3,4,6,7` for `>100` instead of `6`. - `queryParser.test.ts`: 16 new cases (12 numeric, 4 unchanged). - `packages/common-utils`: 3,175 tests pass; `ci:lint` 0 errors, warning count unchanged (80). ### How to test on Vercel preview **Preview routes:** /search **Steps:** 1. Click **Connect to Demo Server** on the onboarding screen. 2. Open /search with the **Demo Logs** source and **Last 15 minutes**. 3. Set SELECT to `Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body`. 4. Search `LogAttributes.infra.gc_pause_ms:[10 TO 20]`. 5. Verify results appear and every `gc_pause_ms` value is between 10 and 20. 6. Search `LogAttributes.infra.gc_pause_ms:>100` and confirm no results are returned. ### References - Related PRs: #2779 (range fixes, verified on numeric columns; its map-attribute range test asserted SQL that fails on a String map), #2841 ### Update: After review, this PR now only fixes range searches like field:[10 TO 20] on text fields. >, <, >= and <= work the same as on main (they compare as text) so existing saved searches and alerts don't change. Use a range like field:[100 TO *] for a numeric comparison. The >100 parts below no longer apply.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A search on a
Mapcolumn whose value type is numeric orBooldouble-escapes the map subscript, so ClickHouse reads it as one identifier instead of a map lookup:Measures.latency_ms:250```Measures``['latency_ms']` = CAST('250', 'Float64')Measures.latency_ms:"250"`Measures`['latency_ms'] = CAST('250', 'Float64')Flags.cached:true```Flags``['cached']` = 1For
Map(String, Float64)only the unquoted spelling is affected — quoting routes through a different branch. ForMap(String, Bool)both are, so a Bool-map search has no working spelling.Map(String, String)and plain columns are unaffected.getColumnForField()returnscolumnalready rendered for a map key and bare for a plain column. Three equality branches (SQLSerializer.eqBool,fieldSearchBool and Number) pass it as a SqlString??placeholder, which appliesescapeIdto it a second time; every sibling branch —eq/Number,gte,lte,lt,gt,range, the finalILIKE— interpolates it raw.Just matching the siblings would drop the backticks a plain Bool or numeric column gets today:
IsError:truewould go from`IsError` = 1toIsError = 1. So the escaping stays and is skipped only when the expression is already a rendered Map subscript, whichgetColumnForField()now reports viamapKey. The issue suggested keying offmapKeyIndexExpressioninstead — that one is also set on the exactMatch path for a materialized column, so it would have stripped the backticks from those. If you'd rather the serializer were uniformly raw and accept that quoting change, say so and I'll switch.Tests: eleven cases in
queryParser.test.tscovering unquoted, quoted and negated numeric-map searches, Bool-map in both directions, and pins that plain and materializedBool/UInt64columns keep their backticks. Five fail onmainand pass with this change; the pins hold either way.make ci-unitandmake ci-lintare green locally. I haven't run this against a live ClickHouse — the evidence is the emitted SQL.Screenshots or video
N/A — non-UI change.
How to test on Vercel preview
N/A — non-UI change.
References