Skip to content

fix(common-utils): merge DDSketch and GK quantile states with their accuracy - #3737

Draft
alex-fedotyev wants to merge 1 commit into
alex/HDX-6120-trace-rollup-1-demo-mv-configfrom
alex/HDX-6120-trace-rollup-2-dd-merge-params
Draft

alex-fedotyev wants to merge 1 commit into
alex/HDX-6120-trace-rollup-1-demo-mv-configfrom
alex/HDX-6120-trace-rollup-2-dd-merge-params

Conversation

@alex-fedotyev

@alex-fedotyev alex-fedotyev commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stack

Layer 2 of 5 in a GitHub stack (stack #3741). GitHub shows each layer's base as the branch below it, but the stack's base is main: merging any PR lands it and every PR below it on main, one squashed commit each. Each PR's diff is only its own layer, and CI runs as if each targets main:

  1. fix(app): match the demo traces rollup's dimension columns to its table #3736 Demo traces rollup lists its real dimension columns
  2. fix(common-utils): merge DDSketch and GK quantile states with their accuracy #3737 DDSketch and GK states merge with their accuracy (this PR)
  3. fix(app): keep search counts exact when a materialized view serves the chart #3738 Search counts stay exact when a view serves them
  4. feat(app): show materialized view acceleration on the trace search RED charts #3739 Acceleration indicator on the trace search RED charts
  5. feat(app): infer a materialized view's minimum date from when it was created #3740 Minimum date inferred from the view's creation time

The layers are independent fixes to how charts read materialized views, ordered so the smallest land first. They prepare for a default trace rollup, which comes in a follow-up stack.

Summary

A view column built with quantileDDState(0.01, 0.5) or quantileGKState(100, 0.5) stores the sketch's accuracy in its state. The optimizer turned a p95 over such a column into quantileDDMerge(0.95)(col). ClickHouse accepts that, but reads 0.95 as the accuracy and uses the default level, so the "p95" came back as roughly the median: 4.5 ms instead of 911 ms over a day of demo spans, with no error. For GK the same call fails the EXPLAIN check, so those views were never used. Alerts go through the same optimizer, so a p95 alert over a DDSketch view compared against the wrong number.

The optimizer now reads the accuracy from the column type and carries it on the select item (sketchParams), and the renderer emits quantileDDMerge(0.01, 0.95)(col). Singular and plural states (quantilesDD(0.01, 0.9, 0.99)) both work. sketchParams is set only by the optimizer at query time: it isn't saved with charts and isn't part of the external API. t-digest, reservoir, BFloat16 and exact sketches don't keep the level in their state, so they render exactly as before and the existing snapshots don't change.

Test plan

  • New integration test (materializedViewQuantileSketches.int.test.ts): builds 1-minute rollups with quantileDD, quantileGK, quantileTDigest and quantileBFloat16 states over 100,000 spans, reads p95 and p99 through the optimizer at 5-minute buckets (so each bucket merges five states), and compares them to quantileExact on the raw table. On main the DDSketch and GK cases fail; with this change all four are within their error bound.
  • Unit tests for the select conversion (quantileDD, quantilesDD, quantileGK, and BFloat16 staying parameterless) and for rendering the merge with and without a condition.
  • common-utils ci:lint, unit tests, and the integration test against ClickHouse 26.10.

Part of HDX-6120.

…ccuracy

A materialized view column built with quantileDD(accuracy, level) or
quantileGK(accuracy, level) was merged as quantileDDMerge(level), which
ClickHouse reads as an accuracy, so p95 came back as a different quantile with
no error. GK merges failed the EXPLAIN check, so those views were skipped.

The optimizer now reads the sketch's accuracy from the column type and passes
it to the renderer, which emits quantileDDMerge(0.01, 0.95). Sketches whose
state doesn't depend on the level (t-digest, reservoir, BFloat16, exact) render
as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 465b9c5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 20, 2026 6:21am UTC
hyperdx-storybook Ready Ready Preview Nov 20, 2026 6:21am UTC

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 470 passed • 1 skipped • 1059s

Status Count
✅ Passed 470
❌ Failed 0
⚠️ Flaky 12
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 465b9c52 Deployed Nov 20, 2026 by vercel[bot]
Preview – hyperdx-storybook — 465b9c52 Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant