Skip to content

feat(alerts): add per-group acknowledgments for grouped alerts - #2696

Closed
oddegen wants to merge 3 commits into
ClickHouse:mainfrom
oddegen:main
Closed

oddegen wants to merge 3 commits into
ClickHouse:mainfrom
oddegen:main

Conversation

@oddegen

@oddegen oddegen commented Jul 21, 2026 •

Copy link
Copy Markdown

Summary

Adds per-group alert acknowledgments for grouped alerts.

Grouped alerts can now mute notifications for one group without muting other
groups or changing the aggregate parent alert state. Alert evaluation and
history writes still reflect the real threshold state for every group.

This also adds a group resume override for parent acknowledgments: if the parent
alert is acknowledged, a user can resume notifications for one specific group
while the rest of the grouped alert remains muted by the parent ack.

Screenshots or video

alert_per_group_hyperdx-2026-07-22_16.25.37.mp4

How to test on Vercel preview

Preview routes: /alerts

Steps:

  1. Open the Alerts page with at least one grouped alert in the Triggered state.
  2. Expand the grouped alert row and verify each current group is listed under the parent alert.
  3. Click Ack on one group, choose a duration, and verify only that group changes to Ack'd.
  4. Click the parent alert Ack, choose a duration, and verify firing groups without explicit overrides show the subtle Muted state.
  5. Open a Muted group menu, click Resume group, and verify that group returns to the normal alerting row state while other parent-muted groups stay muted.

References

@changeset-bot

changeset-bot Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0499409

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Jul 21, 2026

Copy link
Copy Markdown

@oddegen is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@greptile-apps

greptile-apps Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds per-group acknowledgments for grouped alerts. The main changes are:

  • New group silence and group resume state on alerts.
  • API routes and response fields for group-level ack controls.
  • Alert evaluation logic that suppresses notifications per group without changing history state.
  • Alerts UI rows and menus for group ack, parent-muted, and resumed states.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
packages/app/src/components/alerts/AlertAckMenu.tsx Adds the shared acknowledgment menu and handles expired silences before showing an acknowledged state.
packages/app/src/components/alerts/AckAlert.tsx Adds group acknowledgment controls, parent-muted display, and group resume actions.
packages/api/src/tasks/checkAlerts/index.ts Adds group-level notification suppression and group key labeling for alert evaluation.
packages/api/src/routers/api/alerts.ts Adds API routes for group silence, group unsilence, and parent acknowledgment cleanup.

Reviews (3): Last reviewed commit: "Merge branch 'main' into main" | Re-trigger Greptile

@oddegen oddegen changed the title Feat/per group alert ack (#2) feat(alerts): add per-group acknowledgments for grouped alerts Jul 21, 2026
oddegen added 2 commits July 22, 2026 17:21
* feat: add per-group alert silence storage

* feat: add alert group silence api

* feat: enforce per-group alert silences

* feat: expose grouped alert state in responses

* feat: add alert group silence hooks

* fix: align alert group silence hooks

* feat: add grouped alert ack UI

* fix: align grouped alert UI organization

* fix: render grouped alerts properly

* feat: add per-group alert acknowledgments
@oddegen
oddegen marked this pull request as ready for review July 22, 2026 14:24

@jordan-simonovski jordan-simonovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, thanks so much for raising this PR! I've left a few comments for you. Given the size of the change, I'd say that it's fine not to address anything which is not considered to be blocking.

Apologies if the number of comments are a bit overwhelming, I just want to make sure that alerting remains reliable for all users!

| { type: 'alert'; silenced: NonNullable<IAlert['silenced']> }
| { type: 'group'; silenced: IAlertSilencedGroup };

export const getAlertNotificationSuppression = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: extracting suppression into a pure getAlertNotificationSuppression(alert, group, now) is the right seam.

Injectable now, no I/O, no document mutation, so the precedence rules are unit-testable in isolation instead of only through processAlert. This is what makes the rest of the feature reviewable at all. My comment on the test file is about widening the matrix over this function, not about the design.

extraFields: Array<[string, string]>,
groupByLabels: string[],
) => {
const fields = extraFields.map(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (non-blocking): the positional label zip mislabels groups whenever extraFields and groupByLabels aren't 1:1.

const fields = extraFields.map(
  ([key, value], index) => [groupByLabels[index] ?? key, value] as [string, string],
);

Two independently-derived lists zipped by index with no length check. The mismatch is easy to hit: valueColumnNames in getResponseMetadata is every column ClickHouse types as Number, so a numeric groupBy column never reaches extraFields. With groupBy: ["StatusCode", "ServiceName"] and a numeric StatusCode, extraFields is [["ServiceName", "api"]] while groupByLabels[0] is "StatusCode" — so the group is labelled StatusCode:api, naming one dimension and showing another's value.

Worth reading alongside my blocking comment on routers/api/alerts.ts — the dropped column is also why two genuinely distinct groups collapse onto one key, which is what makes acking-by-name unsafe. The dropping is pre-existing; the mislabelling is new here.

const labels =
  groupByLabels.length === extraFields.length ? groupByLabels : [];
if (labels.length === 0 && groupByLabels.length > 0) {
  logger.warn(
    { alertId, columns: extraFields.map(([k]) => k), groupByLabels },
    'groupBy label/column mismatch; falling back to result column names',
  );
}
const fields = extraFields.map(
  ([key, value], index) => [labels[index] ?? key, value] as [string, string],
);

That warn is worth having on its own — right now the mismatch is completely silent.

return;
}

if (notificationSuppression?.type === 'group') {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question (non-blocking): should a per-group ack also suppress that group's recovery notification?

trySendNotification is the single funnel for both ALERT and OK, and this check runs before state is examined — so fire → ack → recover-while-acked means the resolve notification never arrives. That mirrors whole-alert ack behaviour, so it's defensible, but per-group acks will make it routine rather than rare. "Ack this group for 2h" reads to most users as stop paging me, not don't tell me when it's fixed.

If it's intended, worth a line of copy in the ack menu. If not, let OK-state notifications through the group-level check.

: '';
const attributes = hasGroupBy ? Object.fromEntries(extraFields) : {};
const { groupKey, attributes } = hasGroupBy
? buildAlertGroupContext(extraFields, groupByLabels)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking): on deploy, in-flight grouped alerts will re-page for incidents that were already acked, and will never send the recovery notification.

Why this blocks. This is a silent, production-only failure in the one product surface whose entire value is tell me when something is wrong, and only then. Concretely, at the moment this ships:

  • A group that is firing right now loses its fired flag, so it fires again — paging whoever is already holding that incident.
  • The group's history under the old key is never evaluated again, so when the underlying problem clears, no recovery notification is ever sent. The incident silently stays "open" as far as notifications are concerned.
  • A numConsecutiveWindows > 1 group restarts its window count, so the next real breach fires late by up to N intervals.
  • For the full 30-day AlertHistory TTL, the alerts page lists each logical group twice (old key and new key), and acking one does not ack the other — so a user who acks a group still gets paged by its twin.

None of this raises an error, fails a deploy, or writes a log. It presents as alert noise and missed recoveries that nobody will connect back to this change, days later. That's why it's blocking rather than a follow-up: it's cheap to fix now and expensive to diagnose in the wild.

What's happening. The persisted group key used to come from the ClickHouse result column names:

const groupKey = hasGroupBy
  ? extraFields.map(([k, v]) => `${k}:${v}`).join(', ')
  : '';

It now comes from getConfiguredGroupByLabels → getGroupByLabel → each groupBy entry's valueExpression. Those differ in ordinary configs, not just aliased ones — your own new unit test shows it: ResourceAttributes['x'] renders to the column arrayElement(ResourceAttributes, 'x'), so the key flips from arrayElement(ResourceAttributes, 'x'):web to ResourceAttributes['x']:web. Map/attribute access is a common groupBy in HyperDX. AlertHistory.group is the join key for computeHistoryMapKey/previousMap, so every pre-deploy row becomes unreachable.

I can see you knew legacy keys exist — getAlertGroupDisplayName in AlertGroupRows.tsx rewrites arrayElement(Map, 'key'): into the new shape, and there's a test named "normalizes legacy dashboard alert group labels for display while preserving ack payload." But that's browser-side and display-only; it still sends the raw legacy string as the ack payload. There's no server-side counterpart in computeHistoryMapKey, previousMap, getPreviousAlertHistories, or getRecentAlertGroupSummaries.

Fix. Cheapest resolution, and my recommendation: don't change the persisted key at all. Keep it on result column names and apply the friendly labels purely at render time — the UI already has the hook. If the key must change, have buildAlertGroupContext return both forms and fall back to the legacy key in the previous-state lookups and the missing-group resolve branch, or ship a one-time rewrite of the last 30 days of AlertHistory.group.

}): Promise<AlertGroupHistorySummary[]> {
const lookbackMs = limit * ALERT_INTERVAL_TO_MINUTES[interval] * 60 * 1000;

const histories = await AlertHistory.find({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking): this can make the alerts page unusable for an entire team, and it gets worse silently as their telemetry grows.

Why this blocks. GET /api/alerts is the alerts page — the first thing someone opens during an incident. This query now runs once per alert with no ceiling on rows returned, and nothing upstream caps how many distinct groups exist: checkAlerts never sets seriesLimit on alert chart configs, so group cardinality is whatever the telemetry happens to contain. One grouped alert on a naturally high-cardinality dimension (pod name, user ID, trace ID) can have thousands of distinct groups in the lookback window, each carrying up to 20 history entries with their own lastValues arrays.

All of that is pulled into the API process heap and serialised into a single response. So the failure isn't "the page is a bit slow" — it's heap pressure on a shared API process and a response large enough to make the alerts page unusable for everyone on the team, at exactly the moment they need it. It also degrades silently and gradually: this will pass review, pass staging, and fall over first in your largest production tenant, where the diagnosis is hardest.

What's happening.

const histories = await AlertHistory.find({
  alert: new ObjectId(alertId),
  createdAt: { $gte: new Date(Date.now() - lookbackMs) },
  group: { $type: 'string' },
}).sort({ group: 1, createdAt: -1 }).lean();

No .limit(), no cap on distinct groups. The {alert, group, createdAt} index makes this cheap to plan — it does nothing to bound the result set. The per-group limit is applied in JS after every document is already in memory. Note the sibling getRecentAlertHistories directly above uses an aggregation for exactly this reason.

Fix. Mirror the sibling with $group/$slice, or cap both dimensions:

const MAX_ALERT_GROUP_SUMMARIES = 50;
const histories = await AlertHistory.find({ ... })
  .sort({ group: 1, createdAt: -1 })
  .limit(MAX_ALERT_GROUP_SUMMARIES * limit)   // backstop; group:1 sort truncates by group
  .lean();
// ...
return Array.from(groupedHistories.values())
  .sort(...)
  .slice(0, MAX_ALERT_GROUP_SUMMARIES);

Prefer an ordering that keeps ALERT-state groups when the cap bites, and return a truncated flag so the UI can say so. Also worth considering: drop groups from the list endpoint entirely and populate it only on GET /alerts/:id.

})
.optional(),
silenced: AlertSilenceSchema.optional(),
silencedGroups: z.array(AlertSilencedGroupSchema).optional(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (non-blocking): keep silencedGroups off the input schema.

AlertBaseObjectSchema is what alertSchema validates POST/PUT bodies against, so clients can now send silencedGroups on create/update and it will validate. makeAlert() doesn't forward it today, so it's silently dropped — but the day someone spreads the validated body into the persisted document, that's ack forgery that bypasses the futureMutedUntil check entirely.

Cheap to pre-empt now: leave it on AlertsPageItemSchema (line 2163) as a response field and drop it from the base input schema.

group.unsilenced?.parentSilencedAt === parentSilenced?.at;

return (
group.state === 'ALERT' &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick (non-blocking): the muted indicator only shows for ALERT groups, but the server mutes every state.

group.state === 'ALERT' &&
isParentAckActive &&

getAlertNotificationSuppression suppresses regardless of group state, so an OK or PENDING group under an active parent ack is muted with no visual indication. Dropping the state gate makes the UI match the server.

{ label: '24 hours', duration: { hours: 24 } },
];

export function AlertAckMenu({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought (non-blocking): 19 props with mode-dependent defaults is a lot for one component.

acknowledgedButtonIcon, secondarySilencedLabel, resetToInheritedLabel, canUnsilence, unacknowledgedButtonLabel… — each caller assembles a different subset, which means this is really two components sharing a duration picker. Splitting into AlertAckMenu (plain alert ack) and AlertGroupAckMenu (group ack with parent-inheritance semantics) would let each one's props be required and meaningful.

Genuinely a taste call, and the file's still under 300 lines — flagging it because the prop count is the tell, not because it needs doing in this PR.

};
});

describe('getAlertNotificationSuppression', () => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo (non-blocking): widen the precedence matrix — the branches most likely to break are the ones not covered.

First, credit where it's due: ~1780 lines of tests shipped alongside the implementation rather than after it, and the legacy-label display test is what told me you'd already thought about old group keys. This is about specific gaps.

getAlertNotificationSuppression is the heart of the feature and the added tests cover the clean cases. Missing:

  • active parent ack and an active group ack on the same group, asserting { type: 'alert' } — pins parent precedence
  • parent ack present but expired, falling through to the group check → { type: 'group' }
  • a group ack whose own until has passed → undefined
  • empty-string group (the non-grouped path), exercising the !group guards

Highest-value addition given the blocking comment on index.ts:1421: an integration test in checkAlerts.int.test.ts that seeds AlertHistory with legacy-format keys, runs one evaluation, and asserts no spurious resolve and no consecutive-window reset. Whichever behaviour you settle on there, pin it with that test.

@@ -0,0 +1,6 @@
---
'@hyperdx/api': patch
'@hyperdx/app': patch

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Given this is a new feature being added to the app, I'd say this is a minor change, not a patch bump. See: https://semver.org/

@teeohhem

teeohhem commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Closing as stale. Please feel free to re-open if there are new changes.

@teeohhem teeohhem closed this Sep 4, 2026
kodiakhq Bot pushed a commit that referenced this pull request Nov 18, 2026
…hreads (#3464)

## Summary

Today the review bot posts **every** finding it can anchor as an inline thread, at every severity, labelled with its own `critical`/`major`/`minor` self-estimate. Apart from #3138 (triage guidance for the author's agent), the reviewer has not changed since #3024.

This PR adds a separate grading pass that assigns each finding a priority, posts only P0–P2 inline, and folds P3 into the sticky summary. It also stops reworded findings from being re-posted as new threads on later pushes.

## Production today vs this PR

| | production (`main`) | this PR |
|---|---|---|
| Label users see | reviewer's own severity: 🔴 critical · 🟠 major · 🔵 minor | priority only: 🔴 P0/P1 · 🟡 P2 · 🔵 P3 (same icons as deep-review) |
| Inline threads | every anchored finding | P0–P2 |
| Folded into the summary | only `minor` findings that could not be anchored | every P3, anchored or not. Folded, never dropped |
| Grading | none | a second pass (Sonnet, `Read` only) grades the finished findings. It cannot add, drop or reword them |
| Reworded finding on a later push | opens a new thread (dedup keyed on the exact title) | recognised as the same finding |
| Grading fails | n/a | nothing is folded; the review posts exactly as on `main` |

### How a finding's priority is decided (`effectivePriority` in `priority.cjs`)

A finding is folded only when **both** graders call it low, and nothing says it matters today:

- the reviewer itself labelled it `minor`, **and**
- the grading pass put it at P2 or P3, **and**
- the grader doesn't say the harm is reachable in the merged code, **and**
- it doesn't read as security-related.

Neither grader can fold anything on its own: a grader P3 on a finding the reviewer called major or critical is shown as P2. Without a grade, nothing is folded (critical → P0, major → P1, minor → P2).

## Measured against human votes

Data: 👍/👎 reactions maintainers left on this bot's inline comments, covering **75 graded findings across 29 PRs** (53 👍, 22 👎) from 4 maintainers. Two kinds of vote are excluded: reactions cast by bots, and "blanket" votes, where one person thumbed ≥10 findings on a PR without a single 👎. The "this PR" column comes from running this branch's `effectivePriority` over the grading pass's output.

| | production | this PR |
|---|---|---|
| Inline threads | 75 / 75 | **49 / 75** |
| Folded as P3 | 0 | **26 (35%)** |
| 👍 major/critical findings still inline | 17 / 17 | **17 / 17** |
| 👎 findings folded | 0 / 22 | 8 / 22 |
| 👍 findings folded | 0 | 18, all ones the reviewer labelled `minor` |

So, on this sample, about a third fewer inline threads, with no upvoted major/critical finding hidden. The cost is 18 upvoted minor findings moving from inline threads into the folded section of the summary.

## Checked against the gold set

The vote data can only see the severity the bot assigned. It can't tell whether a folded finding was one a human treated as **must-fix**. That's what the gold set measures. It contains human review comments on 30 past PRs (93 items, 29 of them must-fix). The deployed reviewer's findings on those PRs were re-graded with this PR's grading prompt and routed with this PR's rule, reusing the existing match judgements.

| | production | this PR | same rule, no reachable-today floor |
|---|---|---|---|
| Inline threads | 122 (4.1/PR) | **93 (3.1/PR)** | 70 (2.3/PR) |
| Must-fix items the reviewer caught | 10/29 | 10/29 | 10/29 |
| …of those, folded out of the inline threads | 0 | **0** | **3** |

Must-fix recall can't move, because grading never changes what was found. The column to read is the last row. The grader rated 3 of the 10 caught must-fix items P2, and the reviewer had labelled all 3 `minor`, so the minor-P2 fold alone would have hidden them:

- #1410: deep-linked equality/range filters are silently dropped from the filter sidebar
- #2604: OpenAPI spec not regenerated for the new tile type
- #2696: `getRecentAlertGroupSummaries` has no DB-side document cap, unlike its sibling

The reachable-today floor is what keeps all three inline. On this data it is the difference between hiding 0 and 3 of the 10.

## Dedup fix

The cross-push dedup key was `file + exact title`, on the assumption that titles are stable between runs. They aren't. #3039 was reviewed four times in 2.3 hours, and one defect was posted three times under three titles; all three were thumbed down. Restatements are now matched by title-word overlap within the same file: at least 0.4 overlap **and** at least 3 shared content words. The second condition stops short titles like "Missing timeout in fetchUser" and "Missing retry in fetchUser" from merging.

The threshold (0.4) was set from all 140 same-PR, same-file pairs of voted findings. Every pair scoring 0.36 or higher is a restatement. At 0.333, restatements collide with genuinely distinct findings. On #3129, "signs every user out on each container restart" and "still signs sessions with the public secret" are two real defects (the second critical) at exactly that score. So one known reword on #3039 is still missed rather than risk merging those two. Both cases are verbatim test fixtures.

## Limits and what isn't measured yet

- **The gold-set result rests on 10 caught must-fix items** from one reviewer run (Sonnet; production runs Opus). Zero folded is the bar, but each item is worth 10 points.
- **The votes are sparse and uneven.** There are 22 👎, mostly from two people whose thresholds differ ~4×, and only 1 of 197 voted comments has more than one reactor.
- **The reachable-today floor costs volume.** In the vote sample it didn't separate 👍 from 👎 (~30% 👎 either way), and it lowers the fold rate there from 52% to 35%. On the gold set it is what keeps all 3 at-risk must-fix items visible, so the cost buys something.
- **Premise findings.** The grading prompt describes findings that object to the change's premise as "published in their own section". This PR does not build that section; those findings are routed by priority like any other.
- **Cost:** one more Sonnet call per reviewed push that has at least one finding. Not yet measured in CI.
- **This PR's own review runs the old workflow,** because `pull_request_target` loads it from `main`. To exercise the new path before merge, dispatch **Claude Code Review** from this branch against an open PR.

## Test plan

- [x] `node --test .github/scripts/code-review/__tests__/*.test.mjs`: 58/58. The priority rule and the title dedup live in `priority.cjs` and `dedup.cjs`.
- [x] Workflow YAML parses; Prettier and knip clean on the changed files
- [x] Ran the helpers over a real PR's findings and grades (#3258: 7 findings → 4 shown, 3 folded)
- [ ] Dispatch from this branch on 2–3 open PRs and check the inline/folded split
- [x] Gold-set check: 0 of 10 caught must-fix items folded (3 without the reachable-today floor)

## Also tried (not in this PR)

- **The reviewer assigning P0–P3 itself, in a single pass.** Its grades were better calibrated, but it caught 7/29 must-fix items against 10/29 (one run each, so possibly noise).
- **Folding all P2s, not just minor ones.** This hid 61% of comments but lost 4 of the 17 upvoted major/critical findings.
- **Folding on the reviewer's `minor` label alone.** On the gold set it hid 46% of the findings humans treated as merge-blocking.
- **Using `reachableToday` as the main signal.** It didn't distinguish 👍 from 👎 in the votes, so here it only acts as a floor. As a floor, it's what protects must-fix items on the gold set.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-group alert acknowledgment for grouped alerts

3 participants