Repository navigation
fix(common-utils): merge DDSketch and GK quantile states with their accuracy - #3737
Draft
alex-fedotyev wants to merge 1 commit into
Draft
alex-fedotyev wants to merge 1 commit into
alex-fedotyev wants to merge 1 commit into
Conversation
…ccuracy A materialized view column built with quantileDD(accuracy, level) or quantileGK(accuracy, level) was merged as quantileDDMerge(level), which ClickHouse reads as an accuracy, so p95 came back as a different quantile with no error. GK merges failed the EXPLAIN check, so those views were skipped. The optimizer now reads the sketch's accuracy from the column type and passes it to the renderer, which emits quantileDDMerge(0.01, 0.95). Sketches whose state doesn't depend on the level (t-digest, reservoir, BFloat16, exact) render as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 465b9c5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
Contributor
E2E Test Results✅ All tests passed • 470 passed • 1 skipped • 1059s
Tests ran across 4 shards in parallel. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Layer 2 of 5 in a GitHub stack (stack #3741). GitHub shows each layer's base as the branch below it, but the stack's base is
main: merging any PR lands it and every PR below it onmain, one squashed commit each. Each PR's diff is only its own layer, and CI runs as if each targetsmain:The layers are independent fixes to how charts read materialized views, ordered so the smallest land first. They prepare for a default trace rollup, which comes in a follow-up stack.
Summary
A view column built with
quantileDDState(0.01, 0.5)orquantileGKState(100, 0.5)stores the sketch's accuracy in its state. The optimizer turned a p95 over such a column intoquantileDDMerge(0.95)(col). ClickHouse accepts that, but reads 0.95 as the accuracy and uses the default level, so the "p95" came back as roughly the median: 4.5 ms instead of 911 ms over a day of demo spans, with no error. For GK the same call fails theEXPLAINcheck, so those views were never used. Alerts go through the same optimizer, so a p95 alert over a DDSketch view compared against the wrong number.The optimizer now reads the accuracy from the column type and carries it on the select item (
sketchParams), and the renderer emitsquantileDDMerge(0.01, 0.95)(col). Singular and plural states (quantilesDD(0.01, 0.9, 0.99)) both work.sketchParamsis set only by the optimizer at query time: it isn't saved with charts and isn't part of the external API. t-digest, reservoir, BFloat16 and exact sketches don't keep the level in their state, so they render exactly as before and the existing snapshots don't change.Test plan
materializedViewQuantileSketches.int.test.ts): builds 1-minute rollups withquantileDD,quantileGK,quantileTDigestandquantileBFloat16states over 100,000 spans, reads p95 and p99 through the optimizer at 5-minute buckets (so each bucket merges five states), and compares them toquantileExacton the raw table. Onmainthe DDSketch and GK cases fail; with this change all four are within their error bound.quantileDD,quantilesDD,quantileGK, and BFloat16 staying parameterless) and for rendering the merge with and without a condition.ci:lint, unit tests, and the integration test against ClickHouse 26.10.Part of HDX-6120.