Skip to content

fix(app): keep search counts exact when a materialized view serves the chart - #3738

Draft
alex-fedotyev wants to merge 2 commits into
alex/HDX-6120-trace-rollup-2-dd-merge-paramsfrom
alex/HDX-6120-trace-rollup-3-exact-mv-edges
Draft

alex-fedotyev wants to merge 2 commits into
alex/HDX-6120-trace-rollup-2-dd-merge-paramsfrom
alex/HDX-6120-trace-rollup-3-exact-mv-edges

Conversation

@alex-fedotyev

@alex-fedotyev alex-fedotyev commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stack

Layer 3 of 5 in a GitHub stack (stack #3741). GitHub shows each layer's base as the branch below it, but the stack's base is main: merging any PR lands it and every PR below it on main, one squashed commit each. Each PR's diff is only its own layer, and CI runs as if each targets main:

  1. fix(app): match the demo traces rollup's dimension columns to its table #3736 Demo traces rollup lists its real dimension columns
  2. fix(common-utils): merge DDSketch and GK quantile states with their accuracy #3737 DDSketch and GK states merge with their accuracy
  3. fix(app): keep search counts exact when a materialized view serves the chart #3738 Search counts stay exact when a view serves them (this PR)
  4. feat(app): show materialized view acceleration on the trace search RED charts #3739 Acceleration indicator on the trace search RED charts
  5. feat(app): infer a materialized view's minimum date from when it was created #3740 Minimum date inferred from the view's creation time

The layers are independent fixes to how charts read materialized views, ordered so the smallest land first. They prepare for a default trace rollup, which comes in a follow-up stack.

Summary

The search page queries the histogram and the results count with the exact selected range, so the count matches the rows in the results table (#2256). When a source has a materialized view that fits, the optimizer widens that range to whole view buckets, and the counts include rows from outside the selection. On a one-hour window over the demo spans that was 3,111,654 spans instead of 3,059,550, and bursty series drift much more: in one window the error count read 30% high because an error burst sat in the partial first minute. The trace search RED charts run the same queries.

For requests with alignDateRangeToGranularity: false, useQueriedChartConfig now splits the query:

  • The partial first and last chart buckets read the source table with the exact bounds.
  • The whole buckets in between read the view, chunked as before.

Each bucket still comes from a single table, so counts, averages and percentiles all stay exact. The view names an unaliased aggregate after its own merge expression (SUM(...) over its count column, where the source table returns count()), so every chunk takes the first chunk's column names, by position: both queries come from the same select list. The extra raw scan is at most two chart buckets, and a range shorter than one whole bucket reads the source table alone. Aligned charts, such as dashboard tiles, are unchanged.

Test plan

  • Unit tests for the query planner (getQueryWindows): no view, an aligned request, an exact request with partial edges, a range shorter than a bucket, a range on bucket boundaries, an inclusive end on a boundary, and a chunked day with contiguous windows.
  • Hook test: checks which table and range each query hits for an exact hour, that rows come back oldest first, and that the count keeps one column name although the view's chunk returns it under the view's name. Fails without the renaming.
  • Unit tests for the renaming: by position, and left alone when names already match or the shapes differ.
  • On the demo service, edge buckets from the raw table plus the view's interior returned 3,059,550 spans and 907 errors for an unaligned hour, identical to the raw table.
  • app ci:lint and unit tests.

Part of HDX-6120.

…e chart

The search page asks for the exact selected range so the histogram and the
results count match the results table (#2256). When a materialized view served
those queries, the optimizer widened the range to whole view buckets, so
counts included rows from outside the selection: about 1.7% more spans on a
one-hour window on the demo data, and much more for bursty series.

For requests with alignDateRangeToGranularity: false, the partial first and
last chart buckets are now read from the source table with the exact bounds,
and only the whole buckets in between from the view. Each bucket still comes
from a single table, so counts, averages and percentiles all stay exact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 70ea21e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 20, 2026 6:30am UTC
hyperdx-storybook Ready Ready Preview Nov 20, 2026 6:30am UTC

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 474 passed • 1 skipped • 1095s

Status Count
✅ Passed 474
❌ Failed 0
⚠️ Flaky 8
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

…nd its table

A materialized view names an unaliased aggregate after its merge expression
(SUM(...) of the view's count column), while the source table returns count().
With the partial edge buckets read from the source table and the rest from the
view, the same series came back under two names, so the search histogram drew
the view's buckets as the wrong series and the results count missed them.

Chunks now take the first chunk's column names, by position: both queries are
built from the same select list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 70ea21e4 Deployed Nov 20, 2026 by vercel[bot]
Preview – hyperdx-storybook — 70ea21e4 Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant