Repository navigation
fix(api): bound alert history lookback by check interval - #2515
satyasai1916 wants to merge 3 commits into
Conversation
Replace the fixed 7-day getPreviousAlertHistories scan with an interval-based narrow window (with floor and 7-day fallback) to reduce MongoDB index scans on the alert checker hot path. Fixes ClickHouse#2434 Co-authored-by: Cursor <cursoragent@cursor.com>
|
@satyasai1916 is attempting to deploy a commit to the HyperDX Team on Vercel. A member of the Team first needs to authorize it. |
🦋 Changeset detectedLatest commit: eef46f4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Greptile SummaryReplaces the hard-coded 7-day lookback in
Confidence Score: 5/5Safe to merge; the narrow-window optimisation is additive and the fallback preserves existing behaviour for gap-recovery scenarios. The two-phase narrow→fallback logic is correct for all reachable cases: alerts with recent history get one aggregation, alerts with older-than-narrow history fall back correctly via the exists() guard, and alerts with no history at all skip the wide query entirely. The computeAlertHistoryLookbackMs formula is verified by unit tests for every distinct range (floor, mid, cap). No files require special attention. Important Files Changed
Reviews (3): Last reviewed commit: "fix(api): prettier formatting in checkAl..." | Re-trigger Greptile |
Use a cheap exists check before the 7-day fallback so new alerts without any history only run one aggregation per tick. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@satyasai1916 One of the lint jobs is failing, could you please take a look |
Co-authored-by: Cursor <cursoragent@cursor.com>
|
This PR currently has a merge conflict. Please resolve this and then re-add the |
|
Closed due to inactivity. Please feel free to re-open when ready! |
## Summary Every check-alerts tick ran `getPreviousAlertHistories` for every alert. It scanned a fixed 7-day window of `AlertHistory` and reduced it with `$sort` + `$group`/`$first`. MongoDB runs that as a plain `IXSCAN` over the whole window, not a per-group short-circuit. The cost grows with history volume, and the slow-query log fills up (see #2434). This PR ports the fix that already runs in the enterprise build. The code and comments are copied from it nearly verbatim, so the upstream sync into that repo conflicts only where it has its own code. - **Skip check.** One `findOne` per alert on `{alert: 1, createdAt: -1}` fetches the latest non-ERROR history row from the last 7 days. If that row is in the current window, `processAlert` would skip the alert anyway, so no history scan runs. A sentinel entry keeps the existing `shouldSkipAlertCheck` path and its metric. The 7-day bound caps how far the lookup walks past ERROR rows for an alert that keeps failing, and matches how long `main` remembered a non-grouped alert's state. - **Non-grouped alerts** take their previous state from that row and never scan. - **Grouped alerts** scan a window of `MAX_NUM_WINDOWS + 2` (52) intervals, clamped to between 1 and 7 days: - 1m to 15m: 1 day - 30m: 26 hours - 1h: 52 hours - 6h and longer: 7 days The scan is a streamed `find()` on `{alert, createdAt}`, reduced to the latest row per group in code, so memory stays at one batch plus one entry per group. - **Catch-up after a long outage.** If a due grouped alert's latest row is older than its scan window, the scan returns nothing. The provider then reads every row from that last evaluation (one per group, with its group-by values), so the evaluation still starts from it and backfills the missed windows, as on `main`, and each of those groups keeps its previous state. - **`getConsecutiveWindowHistories`** uses `find().lean()` with an in-code sort instead of an aggregation with a group-major `$sort`. - **The provider** now resolves all alerts first, then runs the skip check and the history fetches, in a `loadAlerts()` helper laid out like the enterprise one. - **Evaluator failures.** If an alert's window start cannot be computed during the skip check (its evaluator lookup throws), the alert is treated as due instead of failing the whole tick. `processAlert` then records the error on that alert, as it does today. The enterprise build does not have this guard yet. - **Index cleanup.** The `{alert, group, createdAt}` index declaration is removed, because no query uses it now. OSS does not drop indexes at startup, so existing deployments keep the index; new ones no longer create it. ### Trade-offs - **Fast grouped alerts still scan about 1 day.** The 1-day minimum lets groups whose last row predates a task outage still auto-resolve. For a 1m grouped alert this cuts keys examined about 7×, not down to a handful. - **A failing alert's lookup still walks its ERROR rows.** The `state` filter is not in the index, so for an alert that only writes ERROR rows the skip-check lookup reads up to 7 days of them per tick (about 10,000 for a 1m alert), as `main` already does. Healthy alerts read one index key. - **Some groups are forgotten sooner.** A group stuck in ALERT that writes no new rows (threshold met at 0) now drops out of the previous-state map after the shorter window, not after 7 days. This is the case raised in the issue thread. - **Some comments describe DocumentDB and cite internal ticket IDs.** They were kept verbatim to avoid sync conflicts. ### How to reproduce **Starting point:** `yarn dev` with at least one alert, or `make dev-int`. **On `main`:** 1. Create a 1m alert and let it run for a while. 2. Profile the `alerthistories` aggregate. Every tick runs one aggregate per alert, and `keysExamined` grows with the full 7-day window. **On this branch:** 1. Run the same setup. Non-grouped alerts issue only a one-key `findOne` per tick. Grouped alerts issue one bounded `find` only when they are due. 2. Run the integration tests: - `make dev-int FILE=src/tasks/checkAlerts/__tests__/checkAlerts.int.test.ts` - `make dev-int FILE=src/tasks/checkAlerts/providers/__tests__/default.int.test.ts` A new test runs `explain` on the grouped scan and checks that it uses `alert_1_createdAt_-1` with tight bounds. ### Testing - `packages/api` unit tests pass. - These integration suites pass locally: `checkAlerts`, `providers/default`, `singleInvocationAlert` and `multiChannelAlerts`. - New tests cover: - the lookback calculation - per-group results - ERROR rows being ignored - the window bound - one streamed `find` per alert, with no aggregation - the index plan - the 7-day bound on the skip-check lookup - a grouped alert whose last evaluation is older than its scan window: every group is restored with its values - a grouped alert already evaluated this window is skipped by `processAlert` from the sentinel alone - the skip check: already evaluated, grouped and due, non-grouped and due, no history, and a window start that cannot be computed ### References - Fixes #2434 - Refs #3508 (partial). The grouped scan is capped at 1 day and streamed, so it no longer grows for a week and memory stays bounded. A 1m alert with ~1,300 groups still reads about 1 day of rows per tick, so the scan cost that issue reports is reduced, not removed. - Overlaps #3509, which fixes #3508 with a `{alert, group, createdAt, state}` index and a per-group `DISTINCT_SCAN`. This PR removes the `{alert, group, createdAt}` index declaration and the aggregation, so the two conflict. - Supersedes #2515 (closed)
Summary
getPreviousAlertHistorieswith an interval-based narrow window (5x check interval, 1h floor).Fixes #2434
Test plan
computeAlertHistoryLookbackMsvaluesgetPreviousAlertHistoriessuite + fallback case