Skip to content

fix: Ensure search histogram count matches result table count - #2256

Merged
kodiakhq[bot] merged 3 commits into
mainfrom
drew/fix-histogram-counts
May 11, 2026
Merged

kodiakhq[bot] merged 3 commits into
mainfrom
drew/fix-histogram-counts

Conversation

@pulpdrew

Copy link
Copy Markdown
Contributor

Summary

This PR updates the ChartConfig used for the search page histogram and total result count, so that the count always matches what is shown in the results table.

Previously, there were two reasons the count might not match:

  1. When not in live mode, the chart (and count) were aligned to the granularity, whereas the results table was not. This could cause the total count and histogram to over count the number of results
  2. The histogram was end-exclusive, now it is end inclusive to match the search results table config.

Screenshots or video

Before: alignment resulted in a count that was greater than the number of results in the table

Screenshot 2026-05-11 at 9 08 49 AM

After: Count is correct even when date range is not aligned or exactly matches the queried result's timestamp (due to end inclusivity)

Screenshot 2026-05-11 at 9 08 33 AM Screenshot 2026-05-11 at 9 08 20 AM

How to test on Vercel preview

This can be tested in the preview environment by playing around with the filters and time ranges to get a small (countable) number of results. Or locally by seeding clickhouse with some data at particular timestamps (which is what the new E2E test does)

References

@vercel

vercel Bot commented May 11, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hyperdx-oss Ready Ready Preview, Comment May 11, 2026 7:11pm

Request Review

@changeset-bot

changeset-bot Bot commented May 11, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: bfa2dbc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label May 11, 2026
@github-actions

github-actions Bot commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 3
  • Production lines changed: 36 (+ 199 in test files, excluded from tier calculation)
  • Branch: drew/fix-histogram-counts
  • Author: pulpdrew

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

<!-- claude-code-review -->

PR Review

✅ No critical issues found.

A few minor observations (non-blocking):

  • ℹ️ convertToTimeChartConfig now reads config.dateRangeEndInclusive only when alignDateRangeToGranularity === false. Behavior is preserved for existing callers (default ?? false), and only the new search-page caller exercises the inclusive path — looks correct, but worth double-checking there are no other callers that pass alignDateRangeToGranularity: false without an explicit dateRangeEndInclusive who might depend on the old hardcoded exclusivity for boundary events.
  • ℹ️ histogramTimeChartConfig comment still references the live-mode rationale, but alignDateRangeToGranularity is now unconditional — comment is accurate (it's a side benefit) but could be tightened to lead with the count-correctness reason and treat live-mode as a footnote.
  • ℹ️ The new e2e clickhouseQuery helper duplicates URL/auth handling that already exists in seed-clickhouse.ts's createClickHouseClient. Consider extracting a shared helper to keep auth/host plumbing in one place. Not a blocker.

@github-actions

github-actions Bot commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 169 passed • 3 skipped • 1203s

Status Count
✅ Passed 169
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 3

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

🟡 P2 -- recommended

  • packages/app/src/DBSearchPage.tsx:1494 -- Hardcoding alignDateRangeToGranularity: false (replacing !isLive) removes the prior cache-friendly alignment in non-live mode; users who re-submit the same relative range within a granularity bucket now get fresh react-query and server-side query-cache keys instead of cache hits. The performance reviewer additionally claimed a live-mode regression, but in the old code !isLive was already false while live, so live-mode behavior is unchanged -- verify the non-live impact is acceptable before merging.
    • Fix: Confirm the cache-hit loss is acceptable on the search page; otherwise round the histogram's end timestamp to the nearest second (or bucket) for cache-key purposes while still passing the exact range to ClickHouse.
    • performance, adversarial
  • packages/app/tests/e2e/features/search/total-count-matches-table.spec.ts:21-46 -- clickhouseQuery duplicates the env-var/URL/POST-text-plain client logic already implemented as createClickHouseClient in seed-clickhouse.ts; future protocol or auth changes will need to be applied in two places.
    • Fix: Export a small queryClickHouse(sql) helper from seed-clickhouse.ts (or tests/e2e/utils/) and import it in this spec.
    • maintainability
  • packages/app/src/ChartUtils.tsx:118-126 -- The new dateRangeEndInclusive branching in convertToTimeChartConfig is exercised only by the e2e regression test; the four alignDateRangeToGranularity × dateRangeEndInclusive combinations (especially align=false, end=undefined) have no unit coverage in ChartUtils.test.ts.
    • Fix: Add unit cases covering the new branch -- aligned always emits dateRangeEndInclusive: false; unaligned passes the caller's value through, defaulting to false when undefined.
    • correctness, testing, maintainability, kieran-typescript
  • packages/app/tests/e2e/features/search/total-count-matches-table.spec.ts:120-131 -- waitForTotalCountLoaded only checks that the count text is no longer the loading ellipsis; if seedCountMismatchLogs ever fails silently or the ingestion isn't yet visible the chart renders 0 Results, which the wait passes through immediately, and the toHaveCount(5) / toHaveText('5 Results') failures then look like product regressions rather than seeding flakes.
    • Fix: Before the equality assertions, assert the count is non-zero (e.g. expect(searchPage.totalCountText).not.toHaveText('0 Results')) or assert table row count > 0 first so seeding failures surface distinctly.
    • testing
🔵 P3 nitpicks (10)
  • packages/app/src/ChartUtils.tsx:123 -- The ?? fallback (config.dateRangeEndInclusive ?? false) contradicts the DateRange type's // default true comment; today only DBSearchPage reaches this branch with an explicit override, but future callers passing alignDateRangeToGranularity: false without an explicit dateRangeEndInclusive will silently get end-exclusive behavior.
    • Fix: Either change the fallback to ?? true to match the type doc, or update the DateRange comment to reflect the converter's effective default and add a unit test pinning the chosen semantic.
    • kieran-typescript, adversarial
  • packages/app/src/DBSearchPage.tsx:1460-1462 -- Partial<Pick<BuilderChartConfigWithDateRange, 'groupBy'>> allows DerivedColumn[] | string | undefined but the switch branches only ever assign string | undefined; the wider annotation lets a future contributor silently broaden the value past what downstream consumers handle.
    • Fix: Narrow to { groupBy?: string } to match what the switch can produce.
    • kieran-typescript
  • packages/app/src/DBSearchPage.tsx:1487 -- eventTableSelect: searchedConfig.select ?? undefined is load-bearing because SearchConfig.select is string | null | undefined and the target is string | undefined; the necessity is invisible to a reader who doesn't know the upstream type is nullable.
    • Fix: Add a one-line comment noting that searchedConfig.select may be null and the coercion is required for the satisfies constraint.
    • kieran-typescript
  • packages/app/src/ChartUtils.tsx:123 -- isAligned = config.alignDateRangeToGranularity !== false is consumed exactly once and inverts the polarity of the underlying field name, forcing the reader to translate.
    • Fix: Inline the expression or rename to alignmentDisabled to mirror the === false pattern already used in getTimeChartDateRange.
    • maintainability
  • packages/app/src/DBSearchPage.tsx:1505 -- isLive was dropped from the useMemo deps without an // eslint-disable-next-line react-hooks/exhaustive-deps annotation or rationale; the removal is correct (isLive is no longer read) but a future lint-driven "fix" could restore it and resurrect the bug.
    • Fix: Add a brief inline comment at the dep array stating that isLive is intentionally omitted because the histogram no longer depends on it.
    • maintainability
  • packages/app/src/hooks/useChartConfig.tsx:320-345 -- dateRangeEndInclusive: true only survives on chunk 0 of a chunked query (getGranularityAlignedTimeWindows forces false for the rest); for searches over ~15 minutes the histogram total can transiently disagree with the table count between chunk arrivals even after the loading state clears.
    • Fix: Gate the displayed count on the existing isTotalCountComplete signal in SearchTotalCountChart, or document the steady-state caveat in the changeset.
    • adversarial
  • packages/app/src/DBSearchPage.tsx:1494-1497 -- Zero-width or start > end ranges (e.g. shared URLs) now return any events on the exact boundary instead of always 0; the histogram and total both display a value where they previously displayed nothing.
    • Fix: Either accept the behavior change (matches the table) or short-circuit zero/inverted ranges before building the histogram config.
    • adversarial
  • packages/app/tests/e2e/features/search/total-count-matches-table.spec.ts:49-86 -- seedCountMismatchLogs inserts into E2E_LOGS_TABLE at test time; if seedClickHouse() ever runs concurrently (e.g. a future fixture re-runs the global seed), its TRUNCATE TABLE clears the marker rows mid-test.
    • Fix: Add a WHERE ServiceName NOT LIKE 'count_mismatch_%' guard to clearTestData, or otherwise ensure the global seed cannot race the per-test seed.
    • adversarial
  • packages/app/tests/e2e/features/search/total-count-matches-table.spec.ts:49 -- seedCountMismatchLogs lives inside the spec file; the next test needing a fixed-offset event sequence will likely re-implement it (and a fresh copy of clickhouseQuery with it).
    • Fix: Move the helper into tests/e2e/utils/ or expose it from seed-clickhouse.ts so future specs reuse it.
    • maintainability
  • packages/app/src/DBSearchPage.tsx:1505 -- The dep-array change removes isLive and now the only test of the histogram-config-while-live path is implicit in the existing live-tail e2e suite; there is no regression test confirming the count keeps tracking when live mode is toggled on with the new unaligned config.
    • Fix: Add a focused e2e scenario that toggles live tail and asserts the count remains in sync with the table across a poll cycle.
    • testing

Reviewers (9): correctness, testing, maintainability, project-standards, agent-native, learnings-researcher, performance, adversarial, kieran-typescript.

Testing gaps:

  • No coverage for ranges that straddle a bucket boundary or land exactly on the boundary millisecond -- the most sensitive cases for the end-inclusive flip.
  • No coverage for multi-bucket ranges (the spec only exercises a sub-bucket window inside one 15 s bucket).
  • No coverage for the Traces source path through histogramTimeChartConfig (the groupBy switch has a Trace branch but the e2e only exercises Logs).
  • No coverage for chunked queries (>15 min ranges where dateRangeEndInclusive is downgraded to false for non-leading chunks).

@pulpdrew
pulpdrew force-pushed the drew/fix-histogram-counts branch from ea38aba to b8c2616 Compare May 11, 2026 15:30
@github-actions

github-actions Bot commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

  • ✅ No critical issues found.

Notes (non-blocking):

  • ℹ️ convertToTimeChartConfig in ChartUtils.tsx:119-126 now passes through dateRangeEndInclusive from the caller when alignDateRangeToGranularity: false. Only the search page currently relies on the new branch — DBTimeChart and DBEditTimeChartForm callers don't set dateRangeEndInclusive: true, so behavior is unchanged there. Worth adding a unit test for the new branching in ChartUtils.test.ts:742 to guard the inclusivity logic alongside the new E2E.
  • ℹ️ Non-live mode previously aligned to bucket boundaries (helping query-cache hit rate); always-false will reduce de-dup of histogram queries across re-renders that produce slightly different ranges. Acceptable trade-off, but worth verifying no perceptible regression on large dashboards.
  • ℹ️ E2E injects ${markerService} and integer timestamps directly into a CH INSERT — values are derived from Date.now()/Math.random(), so safe; just flagging since the pattern lives next to seed code that future tests may copy.

@brandon-pereira

Copy link
Copy Markdown
Member

I see the count reporting accuractly, but the historgam still shows empty on very small windows, is that in scope of this change? I was thinking we would reduce the granularity on the graph.

Screenshot 2026-05-11 at 12 56 49 PM

@pulpdrew

Copy link
Copy Markdown
Contributor Author

I see the count reporting accuractly, but the historgam still shows empty on very small windows, is that in scope of this change?

Screenshot 2026-05-11 at 12 56 49 PM

Ah interesting, yeah that appears to be a broader bug where stacked bar charts with a single bucket don't render correctly - you can see in the tooltip that the chart is actually getting the correct data, just not displaying it.

Screenshot 2026-05-11 at 3 00 50 PM

Same happens on chart explorer:

Screenshot 2026-05-11 at 3 02 25 PM

Happy to look at that too as a separate issue but I think #2255 was specifically about the counts not matching.

@kodiakhq
kodiakhq Bot merged commit 6dc5d01 into main May 11, 2026
37 of 41 checks passed
@kodiakhq
kodiakhq Bot deleted the drew/fix-histogram-counts branch May 11, 2026 19:48

This branch was successfully deployed

1 active deployment
Preview — bfa2dbc4 Deployed May 11, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Histogram event count doesn't match table row count for small time ranges

2 participants