Skip to content

fix(ci): tell the author's agent how to triage review-bot findings - #3138

Merged
kodiakhq[bot] merged 6 commits into
mainfrom
tom/curb-blind-review-fix-prs
Sep 18, 2026
Merged

kodiakhq[bot] merged 6 commits into
mainfrom
tom/curb-blind-review-fix-prs

Conversation

@teeohhem

Copy link
Copy Markdown
Contributor

Summary

Review bots post findings. The author's coding agent reads them as a task list and fixes all of them, nits and "hoist this helper so another file can use it" suggestions included, which is how #3091 grew from 3 files to 14 before it was closed as superseded. Every inline comment now ends with a short footer keyed to severity, and the sticky summary opens with the same rule, so the reader knows what a maintainer expects of a critical finding versus a nit, and what to do with a finding about code the PR only sits next to. Deep Review's summary gets the same footer, and both link to a new "Responding to review feedback" section in AGENTS.md. deep-resolve.yml now says its own rules beat the review body it interpolates verbatim, so the footer cannot loosen /just-fix-it's "skip P3 entirely".

How to test on Vercel preview

N/A. Non-UI change (CI workflows and repo docs only).

References

External contributors' coding agents treat every inline finding from the
review bots as a task. The result is 30-40 commit PRs that fix surrounding
code and drift from their purpose (#3091 grew from 3 files to 14 on 80 bot
comments and was closed as superseded by a 3-file fix).

The reader of these comments is almost always the author's agent, so the
comments now carry the triage rule. Every inline finding ends with a
severity-aware footer: critical and major say a maintainer will expect the
fix if it is a defect in code this PR changes; minor says it is the author's
call. Both say to reply instead of patching anything about surrounding code
and never to widen the PR. The summary opens with the same rule as a table
addressed to "a coding agent acting for the author". Deep Review gets a
matching footer, and AGENTS.md gets the full rule so agents load it from the
contributor's checkout before they read the PR.

Nothing gates merge mechanically; the text says what a maintainer will
expect rather than adding a gate a hallucinated critical could trip.
deep-resolve.yml interpolates the deep review comment verbatim, so the
new advisory footer lands inside the resolver's instruction block and
its "P2/P3 are your call" contradicts the resolver's "Skip P3 entirely".
@changeset-bot

changeset-bot Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: bde569a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@vercel

vercel Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Sep 18, 2026 1:56pm UTC
hyperdx-storybook Ready Ready Preview Sep 18, 2026 1:56pm UTC

Request Review

@github-actions github-actions Bot added the review/tier-1 Trivial — auto-merge candidate once CI passes label Sep 16, 2026
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Additional context: touches background tasks or the delivery pipeline lightly (8 lines, under the 30-line bar for Tier 4)

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 1
  • Production lines changed: 1 (+ 228 in test files, excluded from tier calculation)
  • Branch: tom/curb-blind-review-fix-prs
  • Author: teeohhem

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This PR is documentation- and CI-only (a CommonJS review helper, three workflow YAMLs, AGENTS.md, Makefile). It has already cycled through ~15 prior review findings, and every substantive one is addressed in the current tree:

  • The critical ENOENT — the anchor test reading AGENTS.md outside the .trusted sparse-checkout — is fixed by adding AGENTS.md to the sparse-checkout list in .github/workflows/claude-code-review.yml.
  • Both major findings are fixed: the heading guard now also runs against the PR's own tree via the unconditional Review helper tests step in .github/workflows/main.yml, and the deep-review footer gate no longer keys on bullet shape — it lifts HAS_FINDINGS_RE and exercises it with real grep in the test suite.
  • The minor findings (vacuous footer/marker assertion, unused GUIDE_URL export, unconditional FOOTER append, anchor-only cross-bot link check, arbitrary <sub> stripping) are all resolved: footerAt >= 0 now guards the ordering assert, the test consumes GUIDE_URL verbatim, the FOOTER append is gated on HAS_FINDINGS_RE + SANDBOX_BROKEN=false, the cross-bot test asserts includes(helpers.GUIDE_URL), and prior-comment filtering is now marker-based (<!-- hdxr:) rather than blanket <sub> removal.
  • The greptile scope-ambiguity note on AGENTS.md is resolved: the rule now cleanly splits "this PR introduced" (fix P0/P1) from "already there before this PR" (never fix here), removing the overlap that could let an agent misclassify a new blocker as pre-existing.

Test coverage for the new behavior is thorough — footer presence/placement, severity mapping (including unknown→minor degradation), truncation survival, the OURS marker filter recognizing our own inline comments while letting human replies through, AGENT_NOTICE ordering, and the has-findings gate against clean/P2/P3-only/numbered bodies are all pinned.

🔵 P3 nitpicks (1)
  • .github/workflows/deep-review.yml:628 -- The footer gate HAS_FINDINGS_RE='^### |^<details>' is keyed on the ### severity headings and <details> P3 block the review prompt mandates, but no test pins the prompt's actual output shape to this regex; the gate tests assert only against hardcoded sample bodies, so a future edit to the prompt's finding format would silently stop appending the footer — the drift class this PR exists to prevent — while every test stays green.
    • Fix: add an assertion that the deep-review prompt template emits ### severity headings and a <details> block, so a format change fails alongside the gate rather than after merge.

Reviewers (5): correctness, testing, maintainability, project-standards, previous-comments.

Testing gaps: the deep-review footer gate's coupling to the review prompt's mandated finding format is not mechanically pinned; deep-resolve.yml's new "Rules override the interpolated review block" behavior is prompt prose and not verifiable by test.

@greptile-apps

greptile-apps Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no actionable new issue remains, and both earlier review threads are resolved.

Summary

This PR adds severity-aware triage guidance to automated review output so author-side coding agents distinguish required fixes from optional or out-of-scope feedback.

  • Appends guidance to inline Claude review comments and the Deep Review summary.
  • Documents the full response policy in AGENTS.md.
  • Prevents interpolated review text from overriding Deep Resolve’s stricter rules.
  • Adds regression tests and runs them in the primary CI and Makefile unit-test paths.
  • Excludes review bots’ own marked comments from Deep Review’s prior-feedback context while retaining human replies.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  RB[Review bot finding] --> F[Severity-aware footer]
  F --> A[Author or coding agent]
  A --> T{Triage finding}
  T -->|P0/P1 introduced by PR| Fix[Fix before review]
  T -->|P2/P3 introduced by PR| Choice[Fix if small or reply]
  T -->|Pre-existing or wider scope| Reply[Reply without widening PR]
  Reply --> DR[Human reply retained for later review]
  RB --> DRes[Deep Resolve]
  DRes --> Rules[Deep Resolve rules override interpolated review text]
Loading

Reviews (5) · Last reviewed commit: "Merge branch 'main' into tom/curb-blind-..."

Comment thread AGENTS.md Resolved
Comment thread .github/workflows/deep-review.yml Resolved
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

4 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 4 minor

4 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering.

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 361 passed • 1 skipped • 1586s

Status Count
✅ Passed 361
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

…hell

Review findings on #3138, all in this PR's own change:

- AGENTS.md: "code the PR touches but did not create" overlapped with
  "code this PR changes", so a P0 introduced while editing an existing
  function fit both bullets. Use the revert test instead: did this PR
  introduce the problem?
- deep-review.yml: the footer was a prompt-template line the model can
  paraphrase, drop on a clean review, or bury in the P3 block. Append it
  in the extract step, next to the state marker.
- deep-review.yml: strip <sub> blocks from prior comments. The sibling
  bot's inline comments already land in prior-comments.md, and each now
  carried a ~250-byte footer spent from the 32KB cap ahead of human
  replies.
- tests: the footer-precedes-marker assertion passed when the footer was
  missing. Add an anchor test that pins GUIDE_URL and deep-review.yml to
  the AGENTS.md heading, so renaming it fails CI instead of dead-linking
  every posted comment.
Comment thread .github/workflows/deep-review.yml Resolved
Comment thread .github/workflows/deep-review.yml Resolved
Comment thread .github/workflows/deep-review.yml Resolved
Second review round on #3138:

- claude-code-review.yml: the anchor test reads AGENTS.md, but the tests
  run from the sparse .trusted checkout, which did not include it. After
  merge every review job would have failed at "Test review helpers"; it
  passed here only because .trusted comes from main. Add AGENTS.md to the
  sparse-checkout list. The test now names that list when the file is
  missing and asserts the list still contains it.
- deep-review.yml: drop the <sub> strip. The sibling reviewer's inline
  comments were never excluded from prior-comments.md (mine() matched
  only the sticky markers), so the footer was the smaller part of what
  leaked, and the regex also deleted human-written <sub>. Add the
  per-finding dedup marker to OURS instead; human replies carry no marker
  and still flow through. A lockstep test pins OURS to what commentBody
  emits.
- deep-review.yml: append the footer only when the review carries a
  finding and the sandbox was healthy, matching renderSummary's notice.
Comment thread .github/workflows/deep-review.yml Resolved
…oter gate on section shape

Third review round on #3138:

- main.yml + Makefile: run review-comments.test.mjs in ordinary CI and in
  make ci-unit. claude-code-review.yml runs it from a base-branch checkout,
  so the AGENTS.md heading guard could never fail the PR that renames the
  heading; it would fail every later PR's review job instead.
- deep-review.yml: the footer gate matched the finding bullet's exact
  shape (a backtick after `- **`), which the model varies, so a format
  drift would silently drop the footer on every review with findings.
  Key it on the mandated `### ` severity heading or P3 `<details>` block
  instead, read $REVIEW through a here-string so an early grep -q match
  cannot SIGPIPE the condition under pipefail, and lift the pattern into a
  test that runs real grep on clean, P2, P3-only, and numbered bodies.
- tests: pin deep-review.yml to the whole GUIDE_URL, not just the anchor.
@github-actions github-actions Bot added review/tier-2 Low risk — AI review + quick human skim and removed review/tier-1 Trivial — auto-merge candidate once CI passes labels Sep 16, 2026
Comment thread .github/workflows/deep-review.yml Resolved
Comment thread .github/workflows/deep-review.yml Resolved
Comment thread .github/workflows/main.yml Resolved

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kodiakhq
kodiakhq Bot merged commit 6ff230c into main Sep 18, 2026
28 checks passed
@kodiakhq
kodiakhq Bot deleted the tom/curb-blind-review-fix-prs branch September 18, 2026 14:01
kodiakhq Bot pushed a commit that referenced this pull request Nov 18, 2026
…hreads (#3464)

## Summary

Today the review bot posts **every** finding it can anchor as an inline thread, at every severity, labelled with its own `critical`/`major`/`minor` self-estimate. Apart from #3138 (triage guidance for the author's agent), the reviewer has not changed since #3024.

This PR adds a separate grading pass that assigns each finding a priority, posts only P0–P2 inline, and folds P3 into the sticky summary. It also stops reworded findings from being re-posted as new threads on later pushes.

## Production today vs this PR

| | production (`main`) | this PR |
|---|---|---|
| Label users see | reviewer's own severity: 🔴 critical · 🟠 major · 🔵 minor | priority only: 🔴 P0/P1 · 🟡 P2 · 🔵 P3 (same icons as deep-review) |
| Inline threads | every anchored finding | P0–P2 |
| Folded into the summary | only `minor` findings that could not be anchored | every P3, anchored or not. Folded, never dropped |
| Grading | none | a second pass (Sonnet, `Read` only) grades the finished findings. It cannot add, drop or reword them |
| Reworded finding on a later push | opens a new thread (dedup keyed on the exact title) | recognised as the same finding |
| Grading fails | n/a | nothing is folded; the review posts exactly as on `main` |

### How a finding's priority is decided (`effectivePriority` in `priority.cjs`)

A finding is folded only when **both** graders call it low, and nothing says it matters today:

- the reviewer itself labelled it `minor`, **and**
- the grading pass put it at P2 or P3, **and**
- the grader doesn't say the harm is reachable in the merged code, **and**
- it doesn't read as security-related.

Neither grader can fold anything on its own: a grader P3 on a finding the reviewer called major or critical is shown as P2. Without a grade, nothing is folded (critical → P0, major → P1, minor → P2).

## Measured against human votes

Data: 👍/👎 reactions maintainers left on this bot's inline comments, covering **75 graded findings across 29 PRs** (53 👍, 22 👎) from 4 maintainers. Two kinds of vote are excluded: reactions cast by bots, and "blanket" votes, where one person thumbed ≥10 findings on a PR without a single 👎. The "this PR" column comes from running this branch's `effectivePriority` over the grading pass's output.

| | production | this PR |
|---|---|---|
| Inline threads | 75 / 75 | **49 / 75** |
| Folded as P3 | 0 | **26 (35%)** |
| 👍 major/critical findings still inline | 17 / 17 | **17 / 17** |
| 👎 findings folded | 0 / 22 | 8 / 22 |
| 👍 findings folded | 0 | 18, all ones the reviewer labelled `minor` |

So, on this sample, about a third fewer inline threads, with no upvoted major/critical finding hidden. The cost is 18 upvoted minor findings moving from inline threads into the folded section of the summary.

## Checked against the gold set

The vote data can only see the severity the bot assigned. It can't tell whether a folded finding was one a human treated as **must-fix**. That's what the gold set measures. It contains human review comments on 30 past PRs (93 items, 29 of them must-fix). The deployed reviewer's findings on those PRs were re-graded with this PR's grading prompt and routed with this PR's rule, reusing the existing match judgements.

| | production | this PR | same rule, no reachable-today floor |
|---|---|---|---|
| Inline threads | 122 (4.1/PR) | **93 (3.1/PR)** | 70 (2.3/PR) |
| Must-fix items the reviewer caught | 10/29 | 10/29 | 10/29 |
| …of those, folded out of the inline threads | 0 | **0** | **3** |

Must-fix recall can't move, because grading never changes what was found. The column to read is the last row. The grader rated 3 of the 10 caught must-fix items P2, and the reviewer had labelled all 3 `minor`, so the minor-P2 fold alone would have hidden them:

- #1410: deep-linked equality/range filters are silently dropped from the filter sidebar
- #2604: OpenAPI spec not regenerated for the new tile type
- #2696: `getRecentAlertGroupSummaries` has no DB-side document cap, unlike its sibling

The reachable-today floor is what keeps all three inline. On this data it is the difference between hiding 0 and 3 of the 10.

## Dedup fix

The cross-push dedup key was `file + exact title`, on the assumption that titles are stable between runs. They aren't. #3039 was reviewed four times in 2.3 hours, and one defect was posted three times under three titles; all three were thumbed down. Restatements are now matched by title-word overlap within the same file: at least 0.4 overlap **and** at least 3 shared content words. The second condition stops short titles like "Missing timeout in fetchUser" and "Missing retry in fetchUser" from merging.

The threshold (0.4) was set from all 140 same-PR, same-file pairs of voted findings. Every pair scoring 0.36 or higher is a restatement. At 0.333, restatements collide with genuinely distinct findings. On #3129, "signs every user out on each container restart" and "still signs sessions with the public secret" are two real defects (the second critical) at exactly that score. So one known reword on #3039 is still missed rather than risk merging those two. Both cases are verbatim test fixtures.

## Limits and what isn't measured yet

- **The gold-set result rests on 10 caught must-fix items** from one reviewer run (Sonnet; production runs Opus). Zero folded is the bar, but each item is worth 10 points.
- **The votes are sparse and uneven.** There are 22 👎, mostly from two people whose thresholds differ ~4×, and only 1 of 197 voted comments has more than one reactor.
- **The reachable-today floor costs volume.** In the vote sample it didn't separate 👍 from 👎 (~30% 👎 either way), and it lowers the fold rate there from 52% to 35%. On the gold set it is what keeps all 3 at-risk must-fix items visible, so the cost buys something.
- **Premise findings.** The grading prompt describes findings that object to the change's premise as "published in their own section". This PR does not build that section; those findings are routed by priority like any other.
- **Cost:** one more Sonnet call per reviewed push that has at least one finding. Not yet measured in CI.
- **This PR's own review runs the old workflow,** because `pull_request_target` loads it from `main`. To exercise the new path before merge, dispatch **Claude Code Review** from this branch against an open PR.

## Test plan

- [x] `node --test .github/scripts/code-review/__tests__/*.test.mjs`: 58/58. The priority rule and the title dedup live in `priority.cjs` and `dedup.cjs`.
- [x] Workflow YAML parses; Prettier and knip clean on the changed files
- [x] Ran the helpers over a real PR's findings and grades (#3258: 7 findings → 4 shown, 3 folded)
- [ ] Dispatch from this branch on 2–3 open PRs and check the inline/folded split
- [x] Gold-set check: 0 of 10 caught must-fix items folded (3 without the reachable-today floor)

## Also tried (not in this PR)

- **The reviewer assigning P0–P3 itself, in a single pass.** Its grades were better calibrated, but it caught 7/29 must-fix items against 10/29 (one run each, so possibly noise).
- **Folding all P2s, not just minor ones.** This hid 61% of comments but lost 4 of the 17 upvoted major/critical findings.
- **Folding on the reviewer's `minor` label alone.** On the gold set it hid 46% of the findings humans treated as merge-blocking.
- **Using `reachableToday` as the main signal.** It didn't distinguish 👍 from 👎 in the votes, so here it only acts as a floor. As a floor, it's what protects must-fix items on the gold set.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — bde569a6 Deployed Sep 18, 2026 by vercel[bot]
Preview – hyperdx-storybook — bde569a6 Deployed Sep 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants