Skip to content

perf: read a configurable set of parts for text index key and value lookups - #3635

Merged
knudtty merged 10 commits into
claude/dedupe-map-keysfrom
claude/map-key-index-median-parts
Nov 19, 2026
Merged

knudtty merged 10 commits into
claude/dedupe-map-keysfrom
claude/map-key-index-median-parts

Conversation

@knudtty

@knudtty knudtty commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Map key discovery (getMapKeys) and "all values" lookups (getAllKeyValues / streamAllKeyValues) read text indexes through mergeTreeTextIndex, and they read the dictionary of every part in the time range. On tables with many parts, and on Cloud where opening each part's index costs a round trip to object storage, that makes field and filter-value loading slow.

They now read only the 20 parts closest to the median row count. That covers the mapKeys index and the key=value items index for keys, and the key=value and plain-column text indexes for values:

  • Tiny parts that haven't been merged yet hold few keys per part opened.
  • The largest merged parts have the biggest dictionaries; the items index stores one token per distinct key=value pair, so those can be huge.
  • With 20 or fewer parts in the range, every part is read, same as before.

Log and trace sources get two new optional settings, in the source form and on the internal API, external v2 API, and MCP save_source:

  • mapKeyIndexMaxParts ("Map key index parts"): how many parts to read, default 20.
  • mapKeyIndexPartSelection ("Map key index part size"): which parts, by row count. median (default) picks from the middle out, smallest reads the smallest parts (usually the newest, unmerged ones: cheap, and new keys show up first), largest reads the most merged data (covers the most keys, but reads the biggest dictionaries).

Both settings flow from the source through TableConnection / tcFromSource into getMapKeys and getAllKeyValues (filter panel, search bar fields, MCP describe_source), and are part of each lookup's cache key.

"Load more" on a single filter key passes mapKeyIndexMaxParts: null, which reads every part in the time range: it's asking for the complete list, not a sample. null means no part limit; unset means the default of 20.

Checked on ClickHouse 26.10: the part_name IN (…) filter, including the window-function subquery, is applied before the index is read. A one-part limit reads only that part's dictionary.

Trade-off: keys and values that appear only in parts outside the selection can be missed (except via "load more"), and the sum(cardinality) ranking reflects the selected parts instead of total volume. That's marked with a shortcut: comment; raising the part count or choosing other parts for the source fixes it. The table-scan fallback was already row-limited, so this doesn't make results less complete than that path.

Stacked on #3611, which moved the reads into queryMapKeys.

Screenshots or video

Added settings
image

How to reproduce

Starting point: yarn dev (local ClickHouse 26.3+ with the default OTel schema, which has text indexes on mapKeys(...) and the attribute items).

On main:

  1. Open Search on the Logs source and open the filter panel so Map keys load.
  2. In system.query_log, find the mergeTreeTextIndex query: its parts subquery has no limit, so it reads every overlapping part.

On this branch:

  1. Repeat step 1. The mergeTreeTextIndex query's parts subquery now ends with ORDER BY abs(rn - (cnt + 1) / 2), name LIMIT 20.
  2. Team settings → Sources → edit Logs → set "Map key index parts" to 1 and save. Reload Search; the query uses LIMIT 1. Clearing the field goes back to 20.
  3. Set "Map key index part size" to "Largest" and save. Reload Search; the parts subquery orders by rn DESC ("Smallest": rn).

How to test on Vercel preview

Preview routes: /team

Steps:

  1. Open the Logs source under Sources and click to edit it.
  2. Verify a "Map key index parts" number input appears below "Use Text Index", with placeholder 20.
  3. Verify a "Map key index part size" select appears below it, showing "Median (default)", with options "Smallest" and "Largest".

References

@knudtty
knudtty requested a review from a team as a code owner November 12, 2026 19:11
@vercel

vercel Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 19, 2026 3:33pm UTC
hyperdx-storybook Ready Ready Preview Nov 19, 2026 3:33pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f02de10

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 P0/P1 · 🟡 2 P2 · 🔵 0 P3

None could be anchored to changed lines; they are listed below. 1 unchanged from an earlier push (already inline above).

Findings outside the changed lines

  • 🟡 P2 packages/api/src/controllers/ai.ts:111 — AI query assist's getAllFields ignores the source's mapKeyIndexMaxParts / mapKeyIndexPartSelection → This file isn't in the diff, but it is a caller of getAllFields. It takes only .metadataMVs from tcFromSource(source), so Map key discovery here always uses the default (20 parts, median) and ignores what the user set on the source. Spread ...tcFromSource(source) into the getAllFields call, as useMultipleAllFields in packages/app/src/hooks/useMetadata.tsx does with ...tc, so every key lookup respects the per-source settings.

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a well-scoped performance change that threads two new per-source settings (mapKeyIndexMaxParts, mapKeyIndexPartSelection) through the query layer, API, MCP, external v2, and the source form. The high-risk surfaces check out: the UNSAFE_RAW_SQL order string in partsOverlapFilter is built from an exhaustive switch over a validated enum (never caller input), mapKeyIndexMaxParts binds as an Int32 param, invalid values (0, 2.5, -1, unknown enum) are rejected by Zod on every write path and covered by integration tests, the null = "read every part" contract is preserved end-to-end, and the new settings participate in every relevant cache key. Only P3 nits remain.

🔵 P3 nitpicks (4)
  • packages/common-utils/src/core/metadata.ts:1792 — partsOverlapFilter is rebuilt inside the per-column for loop in getMapTextIndexKeyValues (and again near line 1878 in getTextIndexKeyValues) with loop-invariant arguments, so the heavier system.parts window-function subquery is duplicated into every UNION ALL branch.
    • Fix: Hoist the partsOverlapFilter call above the loop and reuse the single ChSql fragment across all branches.
    • performance
  • packages/api/src/mcp/tools/sources/schemas.ts:489 — mapKeyIndexPartSelection uses a hardcoded z.enum(['median','smallest','largest']) instead of deriving from the shared MapKeyIndexPartSelection enum, and omits the compile-time sync assertion the file already applies to SOURCE_KINDS, so it can silently drift from the enum.
    • Fix: Derive the schema from MapKeyIndexPartSelection (e.g. z.nativeEnum) or add an assertion tying the literals to the enum.
    • maintainability, project-standards
  • packages/common-utils/src/types.ts:2879 — OpenAPI documents mapKeyIndexPartSelection as nullable: true, but its Zod schema is .optional() with no null handling, so an external client sending an explicit null gets a 400, while the sibling mapKeyIndexMaxParts preprocesses null to undefined and accepts it.
    • Fix: Either preprocess null to undefined for mapKeyIndexPartSelection or drop nullable: true from its OpenAPI docs so the two fields handle null consistently.
    • api-contract
  • packages/app/src/components/Sources/SourceForm/MapKeyIndexMaxPartsFormRow.tsx:48 — the selection field is labeled "Map key index part size", implying a size value, but the control picks which parts by row-count ranking (median / smallest / largest).
    • Fix: Rename the label to reflect part selection by row count rather than a size.
    • maintainability

Reviewers (12): correctness, adversarial, security, testing, maintainability, project-standards, performance, api-contract, kieran-typescript, previous-comments, agent-native, learnings-researcher.

Testing gaps: (core SQL generation, cache-key variation, boundary/enum rejection, and the null="all parts" path are well covered; the gaps below are peripheral passthrough)

  • describe_source propagation of the new settings into getMapKeys/getAllKeyValues is not asserted — the test change only unbreaks the mock by importing the real tcFromSource.
  • MCP save_source persistence and validation of the two new fields is untested (the REST router path is covered).
  • External v2 API round-trip of the two new fields is untested.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@knudtty
knudtty force-pushed the claude/map-key-index-median-parts branch from c734947 to 507596c Compare November 12, 2026 19:24
@knudtty
knudtty changed the base branch from claude/dedupe-map-keys to claude/topk-text-index-values November 12, 2026 19:24
@knudtty
knudtty added this pull request to stack #3619 November 12, 2026 19:24
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

4 new clones introduced by this PR.

  • packages/api/src/models/source.ts:180-206 ↔ packages/api/src/models/source.ts:237-263 — 27 lines
  • packages/common-utils/src/core/metadata.ts:1271-1295 ↔ packages/common-utils/src/core/metadata.ts:1319-1340 — 25 lines
  • packages/common-utils/src/core/metadata.ts:1765-1774 ↔ packages/common-utils/src/core/metadata.ts:1864-1873 — 10 lines
  • packages/common-utils/src/types.ts:2871-2885 ↔ packages/common-utils/src/types.ts:2920-2934 — 15 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 482 passed • 1 skipped • 1195s

Status Count
✅ Passed 482
❌ Failed 0
⚠️ Flaky 0
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@knudtty knudtty changed the title perf: read only median-sized parts when discovering Map keys from a text index perf: read a configurable set of parts when discovering Map keys from a text index Nov 12, 2026
@knudtty
knudtty removed this pull request from stack #3638 November 13, 2026 01:05
@knudtty
knudtty added this pull request to stack #3643 November 13, 2026 01:05
@knudtty knudtty changed the title perf: read a configurable set of parts when discovering Map keys from a text index perf: read a configurable set of parts for text index key and value lookups Nov 13, 2026
Comment thread packages/common-utils/src/types.ts Resolved
@knudtty
knudtty force-pushed the claude/map-key-index-median-parts branch from 5b42a49 to f02de10 Compare November 19, 2026 15:30
@knudtty
knudtty merged commit 86acf27 into main Nov 19, 2026
28 checks passed

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — f02de102 Deployed Nov 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — f02de102 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants