Skip to content

fix(common-utils): escape Map subscripts once in numeric and Bool searches - #2841

Merged
kodiakhq[bot] merged 3 commits into
ClickHouse:mainfrom
shuvamk:claude/escape-map-subscript-once
Aug 10, 2026
Merged

kodiakhq[bot] merged 3 commits into
ClickHouse:mainfrom
shuvamk:claude/escape-map-subscript-once

Conversation

@shuvamk

@shuvamk shuvamk commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

A search on a Map column whose value type is numeric or Bool double-escapes the map subscript, so ClickHouse reads it as one identifier instead of a map lookup:

Search Emitted predicate
Measures.latency_ms:250 ```Measures``['latency_ms']` = CAST('250', 'Float64')
Measures.latency_ms:"250" `Measures`['latency_ms'] = CAST('250', 'Float64')
Flags.cached:true ```Flags``['cached']` = 1

For Map(String, Float64) only the unquoted spelling is affected — quoting routes through a different branch. For Map(String, Bool) both are, so a Bool-map search has no working spelling. Map(String, String) and plain columns are unaffected.

getColumnForField() returns column already rendered for a map key and bare for a plain column. Three equality branches (SQLSerializer.eq Bool, fieldSearch Bool and Number) pass it as a SqlString ?? placeholder, which applies escapeId to it a second time; every sibling branch — eq/Number, gte, lte, lt, gt, range, the final ILIKE — interpolates it raw.

Just matching the siblings would drop the backticks a plain Bool or numeric column gets today: IsError:true would go from `IsError` = 1 to IsError = 1. So the escaping stays and is skipped only when the expression is already a rendered Map subscript, which getColumnForField() now reports via mapKey. The issue suggested keying off mapKeyIndexExpression instead — that one is also set on the exactMatch path for a materialized column, so it would have stripped the backticks from those. If you'd rather the serializer were uniformly raw and accept that quoting change, say so and I'll switch.

Tests: eleven cases in queryParser.test.ts covering unquoted, quoted and negated numeric-map searches, Bool-map in both directions, and pins that plain and materialized Bool/UInt64 columns keep their backticks. Five fail on main and pass with this change; the pins hold either way. make ci-unit and make ci-lint are green locally. I haven't run this against a live ClickHouse — the evidence is the emitted SQL.

Screenshots or video

N/A — non-UI change.

How to test on Vercel preview

N/A — non-UI change.

References

…rches

Searching a Map column whose value type is numeric or Bool wrapped the map
subscript in a second layer of backticks, so ClickHouse read it as a single
identifier rather than a map lookup:

  Measures.latency_ms:250  ->  ```Measures``['latency_ms']` = CAST('250', 'Float64')
  Flags.cached:true        ->  ```Flags``['cached']` = 1

getColumnForField() returns `column` already rendered for a map key
(`Measures`['latency_ms']) and bare for a plain column. Three equality
branches passed it as a SqlString `??` placeholder, which applies escapeId to
it again; every sibling branch (eq/Number, gte, lte, lt, gt, range, ILIKE)
interpolates it raw. For Map(String, Float64) only the unquoted spelling was
affected, since the quoted form routes through eq/Number; for
Map(String, Bool) both spellings were.

Interpolating raw at all three sites would have dropped the backticks a plain
Bool or numeric column gets today - `IsError` = 1 would become IsError = 1 -
so the escaping is kept and skipped only when the expression is already a
rendered Map subscript, which getColumnForField now reports via mapKey.

Fixes ClickHouse#2765
@changeset-bot

changeset-bot Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 192c766

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@hyperdx/common-utils Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 8, 2026

Copy link
Copy Markdown

@shuvamk is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@greptile-apps

greptile-apps Bot commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR corrects numeric and Bool Map equality serialization by preserving already-rendered Map subscripts while retaining identifier escaping for ordinary and materialized columns.

  • Adds mapKey to resolved field metadata and uses it to distinguish Map expressions from plain identifiers.
  • Adds regression cases for quoted, unquoted, and negated Map searches and for plain/materialized columns.
  • Adds a patch changeset for @hyperdx/common-utils.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
packages/common-utils/src/queryParser.ts Carries Map-key metadata into equality serialization and avoids applying identifier escaping twice to pre-rendered Map subscripts.
packages/common-utils/src/tests/queryParser.test.ts Adds focused SQL-output regressions for numeric and Bool Maps while pinning plain and materialized column escaping.
.changeset/map-subscript-double-escape.md Records the query-serialization fix as a patch release for the affected package.

Reviews (3): Last reviewed commit: "Merge branch 'main' into claude/escape-m..." | Re-trigger Greptile

@kodiakhq
kodiakhq Bot merged commit 2d33b83 into ClickHouse:main Aug 10, 2026
23 of 26 checks passed
kodiakhq Bot pushed a commit that referenced this pull request Nov 18, 2026
#3693)

## Summary

Attribute maps in the ClickStack schema store every value as a `String`, so Lucene comparisons on attributes compiled to string comparisons. `LogAttributes.gc_pause_ms:>100` became `LogAttributes['gc_pause_ms'] > '100'`, which matches `'17.25'` and `'5'` and misses `'150'`. Ranges with numeric bounds became `... BETWEEN 10 AND 20` on a String, which ClickHouse rejects ("no supertype for types String, UInt8").

When the field is String-typed and every bound is a numeric literal, `gt`, `gte`, `lt`, `lte` and `range` now compare `toFloat64OrNull(field)`:

```
LogAttributes.gc_pause_ms:>100
before: (`LogAttributes`['gc_pause_ms'] > '100' AND indexHint(...))
after:  (ifNull(toFloat64OrNull(`LogAttributes`['gc_pause_ms']) > 100, 0) AND indexHint(...))
```

- Values that aren't numbers convert to NULL and don't match. `ifNull(..., 0)` keeps that a plain non-match instead of NULL, so `NOT field:>100` still includes rows where the attribute is missing or non-numeric, as it does today.
- Unchanged: numeric columns (`Duration:>100`), JSON paths, and non-numeric bounds (`version:>abc`, hex such as `0x10`), which still compare as strings.
- It applies to any String-typed field, not only map subscripts, so an attribute behaves the same whether it is read from the map or from a materialized String column.
- The `indexHint(mapContains(...))` postfix is kept.

The existing tests mocked `LogAttributes` as a bare `Map` and only asserted SQL text, which is how `BETWEEN 1 AND 5` on a String passed without ever running. One existing case, `foo:[1 TO 5]` on a String column, now expects the numeric form; its old expected SQL fails on ClickHouse.

Closes #3692

### Screenshots or video

Same live demo data, Demo Logs, last 15 minutes. `infra.gc_pause_ms` peaks at 50 in this data.

| Before (`main`) | After (this branch) |
| :----- | :---- |
| `gc_pause_ms:>100`: no error, but 54,841 rows whose values (8.1, 16.13, 32…) are all below 100 | 0 results |
| <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/05018a18-3add-4935-ac09-b4ac7f9f5607" /> | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/28e6de55-dcc0-45c4-880f-308c7af1ccf5" /> |
| `gc_pause_ms:[10 TO 20]`: "no supertype for types String, UInt8" | 11,668 results, all between 10 and 20 inclusive |
| <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/d942a870-ddd7-410d-a2e7-342c28944424" /> | <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/310bb202-f043-4a54-9675-3611cb029f97" /> |

### How to reproduce

**Starting point:** `yarn app:dev:local`, onboarding → **Connect to Demo Server**.

**On `main`:**

1. Open **Search**, source **Demo Logs**, time range **Last 15 minutes**.
2. Set SELECT to `Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body`.
3. Search `LogAttributes.infra.gc_pause_ms:>100`: rows come back with values below 100.
4. Search `LogAttributes.infra.gc_pause_ms:[10 TO 20]`: the query fails with the "no supertype" error.

**On this branch:**

1. Same steps.
2. Step 3 returns no rows (the demo data never exceeds 50).
3. Step 4 returns rows whose `gc_pause_ms` is between 10 and 20.

### Local validation

- Ran the generated SQL on ClickHouse 26.8 against values `5, 10, 17.25, 20, 100, 150, -3, abc` and a row without the key. Every form (`>`, `>=`, `<`, `<=`, negative bounds, `[a TO b]`, `{a TO b}`, `[a TO *]`, `-field:[a TO b]`, `NOT field:>x`) returned exactly the expected rows. The old SQL returned `1,3,4,6,7` for `>100` instead of `6`.
- `queryParser.test.ts`: 16 new cases (12 numeric, 4 unchanged).
- `packages/common-utils`: 3,175 tests pass; `ci:lint` 0 errors, warning count unchanged (80).

### How to test on Vercel preview

**Preview routes:** /search

**Steps:**

1. Click **Connect to Demo Server** on the onboarding screen.
2. Open /search with the **Demo Logs** source and **Last 15 minutes**.
3. Set SELECT to `Timestamp, ServiceName, LogAttributes['infra.gc_pause_ms'] AS gc_pause_ms, Body`.
4. Search `LogAttributes.infra.gc_pause_ms:[10 TO 20]`.
5. Verify results appear and every `gc_pause_ms` value is between 10 and 20.
6. Search `LogAttributes.infra.gc_pause_ms:>100` and confirm no results are returned.

### References

- Related PRs: #2779 (range fixes, verified on numeric columns; its map-attribute range test asserted SQL that fails on a String map), #2841

### Update: 
After review, this PR now only fixes range searches like field:[10 TO 20] on text fields. >, <, >= and <= work the same as on main (they compare as text) so existing saved searches and alerts don't change. Use a range like field:[100 TO *] for a numeric comparison. The >100 parts below no longer apply.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants