Skip to content

fix(api): bound alert history lookback by check interval - #2515

Closed
satyasai1916 wants to merge 3 commits into
ClickHouse:mainfrom
satyasai1916:fix/2434-alert-history-lookback
Closed

satyasai1916 wants to merge 3 commits into
ClickHouse:mainfrom
satyasai1916:fix/2434-alert-history-lookback

Conversation

@satyasai1916

Copy link
Copy Markdown
Contributor

Summary

  • Replace the fixed 7-day lookback in getPreviousAlertHistories with an interval-based narrow window (5x check interval, 1h floor).
  • Fall back to the existing 7-day window only when the narrow query returns no rows, preserving behavior after checker gaps.
  • Pass each alert's interval from the default provider so lookback scales per alert.

Fixes #2434

Test plan

  • Unit tests: computeAlertHistoryLookbackMs values
  • Integration tests: existing getPreviousAlertHistories suite + fallback case
  • CI: lint, unit, integration

Replace the fixed 7-day getPreviousAlertHistories scan with an
interval-based narrow window (with floor and 7-day fallback) to
reduce MongoDB index scans on the alert checker hot path.

Fixes ClickHouse#2434

Co-authored-by: Cursor <cursoragent@cursor.com>
@vercel

vercel Bot commented Jun 24, 2026

Copy link
Copy Markdown

@satyasai1916 is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@changeset-bot

changeset-bot Bot commented Jun 24, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: eef46f4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@greptile-apps

greptile-apps Bot commented Jun 24, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

Replaces the hard-coded 7-day lookback in getPreviousAlertHistories with a dynamic narrow window (5× check interval, floored at 1 h, capped at 7 d), then falls back to the 7-day window only when the narrow query returns nothing and AlertHistory.exists() confirms older history actually exists — cutting MongoDB index scans on the hot path.

  • index.ts: Extracts the aggregation pipeline into aggregatePreviousAlertHistoriesForAlert, adds computeAlertHistoryLookbackMs, introduces AlertHistoryLookup type, and wires the two-phase narrow→fallback logic into getPreviousAlertHistories.
  • default.ts: Passes each alert's interval through to the query helper, a one-line plumbing change.
  • checkAlerts.test.ts: Adds unit tests for computeAlertHistoryLookbackMs and two integration tests covering the no-history (narrow only) and gap-recovery (fallback) branches; updates all existing call-sites to use the new AlertHistoryLookup shape.

Confidence Score: 5/5

Safe to merge; the narrow-window optimisation is additive and the fallback preserves existing behaviour for gap-recovery scenarios.

The two-phase narrow→fallback logic is correct for all reachable cases: alerts with recent history get one aggregation, alerts with older-than-narrow history fall back correctly via the exists() guard, and alerts with no history at all skip the wide query entirely. The computeAlertHistoryLookbackMs formula is verified by unit tests for every distinct range (floor, mid, cap).

No files require special attention.

Important Files Changed

Filename Overview
packages/api/src/tasks/checkAlerts/index.ts Introduces computeAlertHistoryLookbackMs, aggregatePreviousAlertHistoriesForAlert, and the two-phase narrow→exists→fallback logic; all edge cases (no history, gap recovery, equal-to-MAX guard) handle correctly.
packages/api/src/tasks/checkAlerts/tests/checkAlerts.test.ts Adds computeAlertHistoryLookbackMs unit tests and two integration tests (no-history / fallback). Existing call-sites updated to pass AlertHistoryLookup.
packages/api/src/tasks/checkAlerts/providers/default.ts Passes each alert's interval into getPreviousAlertHistories — straightforward plumbing, no logic change.
.changeset/fix-alert-history-lookback.md Patch-level changeset entry for @hyperdx/api; description matches the implementation.

Reviews (3): Last reviewed commit: "fix(api): prettier formatting in checkAl..." | Re-trigger Greptile

Use a cheap exists check before the 7-day fallback so new alerts
without any history only run one aggregation per tick.

Co-authored-by: Cursor <cursoragent@cursor.com>
fleon
fleon previously approved these changes Jun 29, 2026
@fleon

fleon commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@satyasai1916 One of the lint jobs is failing, could you please take a look

/home/runner/work/hyperdx/hyperdx/packages/api/src/tasks/checkAlerts/__tests__/checkAlerts.test.ts
    8974:24  error  Replace `⏎······ids:·string[],⏎······interval:·AlertInterval·=·'5m',` with `ids:·string[],·interval:·AlertInterval·=·'5m')·=>`  prettier/prettier
    8977:5   error  Replace `)·=>` with `·`                                                                                                         prettier/prettier

Co-authored-by: Cursor <cursoragent@cursor.com>
@satyasai1916

Copy link
Copy Markdown
Contributor Author

@fleon The lint failure is fixed in 87c706e — collapsed the toLookups helper to a single line per prettier. CI should be green on the next run.

@kodiakhq kodiakhq Bot removed the automerge label Jul 6, 2026
@kodiakhq

kodiakhq Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

This PR currently has a merge conflict. Please resolve this and then re-add the automerge label.

@teeohhem

Copy link
Copy Markdown
Contributor

Closed due to inactivity. Please feel free to re-open when ready!

@teeohhem teeohhem closed this Aug 18, 2026
kodiakhq Bot pushed a commit that referenced this pull request Nov 19, 2026
## Summary

Every check-alerts tick ran `getPreviousAlertHistories` for every alert. It scanned a fixed 7-day window of `AlertHistory` and reduced it with `$sort` + `$group`/`$first`. MongoDB runs that as a plain `IXSCAN` over the whole window, not a per-group short-circuit. The cost grows with history volume, and the slow-query log fills up (see #2434).

This PR ports the fix that already runs in the enterprise build. The code and comments are copied from it nearly verbatim, so the upstream sync into that repo conflicts only where it has its own code.

- **Skip check.** One `findOne` per alert on `{alert: 1, createdAt: -1}` fetches the latest non-ERROR history row from the last 7 days. If that row is in the current window, `processAlert` would skip the alert anyway, so no history scan runs. A sentinel entry keeps the existing `shouldSkipAlertCheck` path and its metric. The 7-day bound caps how far the lookup walks past ERROR rows for an alert that keeps failing, and matches how long `main` remembered a non-grouped alert's state.
- **Non-grouped alerts** take their previous state from that row and never scan.
- **Grouped alerts** scan a window of `MAX_NUM_WINDOWS + 2` (52) intervals, clamped to between 1 and 7 days:
  - 1m to 15m: 1 day
  - 30m: 26 hours
  - 1h: 52 hours
  - 6h and longer: 7 days

  The scan is a streamed `find()` on `{alert, createdAt}`, reduced to the latest row per group in code, so memory stays at one batch plus one entry per group.
- **Catch-up after a long outage.** If a due grouped alert's latest row is older than its scan window, the scan returns nothing. The provider then reads every row from that last evaluation (one per group, with its group-by values), so the evaluation still starts from it and backfills the missed windows, as on `main`, and each of those groups keeps its previous state.
- **`getConsecutiveWindowHistories`** uses `find().lean()` with an in-code sort instead of an aggregation with a group-major `$sort`.
- **The provider** now resolves all alerts first, then runs the skip check and the history fetches, in a `loadAlerts()` helper laid out like the enterprise one.
- **Evaluator failures.** If an alert's window start cannot be computed during the skip check (its evaluator lookup throws), the alert is treated as due instead of failing the whole tick. `processAlert` then records the error on that alert, as it does today. The enterprise build does not have this guard yet.
- **Index cleanup.** The `{alert, group, createdAt}` index declaration is removed, because no query uses it now. OSS does not drop indexes at startup, so existing deployments keep the index; new ones no longer create it.

### Trade-offs

- **Fast grouped alerts still scan about 1 day.** The 1-day minimum lets groups whose last row predates a task outage still auto-resolve. For a 1m grouped alert this cuts keys examined about 7×, not down to a handful.
- **A failing alert's lookup still walks its ERROR rows.** The `state` filter is not in the index, so for an alert that only writes ERROR rows the skip-check lookup reads up to 7 days of them per tick (about 10,000 for a 1m alert), as `main` already does. Healthy alerts read one index key.
- **Some groups are forgotten sooner.** A group stuck in ALERT that writes no new rows (threshold met at 0) now drops out of the previous-state map after the shorter window, not after 7 days. This is the case raised in the issue thread.
- **Some comments describe DocumentDB and cite internal ticket IDs.** They were kept verbatim to avoid sync conflicts.

### How to reproduce

**Starting point:** `yarn dev` with at least one alert, or `make dev-int`.

**On `main`:**

1. Create a 1m alert and let it run for a while.
2. Profile the `alerthistories` aggregate. Every tick runs one aggregate per alert, and `keysExamined` grows with the full 7-day window.

**On this branch:**

1. Run the same setup. Non-grouped alerts issue only a one-key `findOne` per tick. Grouped alerts issue one bounded `find` only when they are due.
2. Run the integration tests:
   - `make dev-int FILE=src/tasks/checkAlerts/__tests__/checkAlerts.int.test.ts`
   - `make dev-int FILE=src/tasks/checkAlerts/providers/__tests__/default.int.test.ts`

   A new test runs `explain` on the grouped scan and checks that it uses `alert_1_createdAt_-1` with tight bounds.

### Testing

- `packages/api` unit tests pass.
- These integration suites pass locally: `checkAlerts`, `providers/default`, `singleInvocationAlert` and `multiChannelAlerts`.
- New tests cover:
  - the lookback calculation
  - per-group results
  - ERROR rows being ignored
  - the window bound
  - one streamed `find` per alert, with no aggregation
  - the index plan
  - the 7-day bound on the skip-check lookup
  - a grouped alert whose last evaluation is older than its scan window: every group is restored with its values
  - a grouped alert already evaluated this window is skipped by `processAlert` from the sentinel alone
  - the skip check: already evaluated, grouped and due, non-grouped and due, no history, and a window start that cannot be computed

### References

- Fixes #2434
- Refs #3508 (partial). The grouped scan is capped at 1 day and streamed, so it no longer grows for a week and memory stays bounded. A 1m alert with ~1,300 groups still reads about 1 day of rows per tick, so the scan cost that issue reports is reduced, not removed.
- Overlaps #3509, which fixes #3508 with a `{alert, group, createdAt, state}` index and a per-group `DISTINCT_SCAN`. This PR removes the `{alert, group, createdAt}` index declaration and the aggregation, so the two conflict.
- Supersedes #2515 (closed)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

checkAlerts: getPreviousAlertHistories scans a fixed 7-day window per alert per tick (only needs the latest record per group)

4 participants