Repository navigation
perf: read a configurable set of parts for text index key and value lookups - #3635
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: f02de10 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
PR Review
2 finding(s): 🔴 0 P0/P1 · 🟡 2 P2 · 🔵 0 P3 None could be anchored to changed lines; they are listed below. 1 unchanged from an earlier push (already inline above). Findings outside the changed lines
Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides. |
Deep Review✅ No critical issues found. This is a well-scoped performance change that threads two new per-source settings ( 🔵 P3 nitpicks (4)
Reviewers (12): correctness, adversarial, security, testing, maintainability, project-standards, performance, api-contract, kieran-typescript, previous-comments, agent-native, learnings-researcher. Testing gaps: (core SQL generation, cache-key variation, boundary/enum rejection, and the
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
c734947 to
507596c
Compare
Duplicate code analysis4 new clones introduced by this PR.
If one of these duplicates a helper that already exists, import it instead jscpd compared this branch against |
507596c to
f09f916
Compare
E2E Test Results✅ All tests passed • 482 passed • 1 skipped • 1195s
Tests ran across 4 shards in parallel. |
fcd35ac to
5b42a49
Compare
5b42a49 to
f02de10
Compare
Summary
Map key discovery (
getMapKeys) and "all values" lookups (getAllKeyValues/streamAllKeyValues) read text indexes throughmergeTreeTextIndex, and they read the dictionary of every part in the time range. On tables with many parts, and on Cloud where opening each part's index costs a round trip to object storage, that makes field and filter-value loading slow.They now read only the 20 parts closest to the median row count. That covers the
mapKeysindex and the key=value items index for keys, and the key=value and plain-column text indexes for values:key=valuepair, so those can be huge.Log and trace sources get two new optional settings, in the source form and on the internal API, external v2 API, and MCP
save_source:mapKeyIndexMaxParts("Map key index parts"): how many parts to read, default 20.mapKeyIndexPartSelection("Map key index part size"): which parts, by row count.median(default) picks from the middle out,smallestreads the smallest parts (usually the newest, unmerged ones: cheap, and new keys show up first),largestreads the most merged data (covers the most keys, but reads the biggest dictionaries).Both settings flow from the source through
TableConnection/tcFromSourceintogetMapKeysandgetAllKeyValues(filter panel, search bar fields, MCPdescribe_source), and are part of each lookup's cache key."Load more" on a single filter key passes
mapKeyIndexMaxParts: null, which reads every part in the time range: it's asking for the complete list, not a sample.nullmeans no part limit; unset means the default of 20.Checked on ClickHouse 26.10: the
part_name IN (…)filter, including the window-function subquery, is applied before the index is read. A one-part limit reads only that part's dictionary.Trade-off: keys and values that appear only in parts outside the selection can be missed (except via "load more"), and the
sum(cardinality)ranking reflects the selected parts instead of total volume. That's marked with ashortcut:comment; raising the part count or choosing other parts for the source fixes it. The table-scan fallback was already row-limited, so this doesn't make results less complete than that path.Stacked on #3611, which moved the reads into
queryMapKeys.Screenshots or video
Added settings

How to reproduce
Starting point:
yarn dev(local ClickHouse 26.3+ with the default OTel schema, which has text indexes onmapKeys(...)and the attribute items).On
main:system.query_log, find themergeTreeTextIndexquery: its parts subquery has no limit, so it reads every overlapping part.On this branch:
mergeTreeTextIndexquery's parts subquery now ends withORDER BY abs(rn - (cnt + 1) / 2), name LIMIT 20.LIMIT 1. Clearing the field goes back to 20.rn DESC("Smallest":rn).How to test on Vercel preview
Preview routes: /team
Steps:
References