Repository navigation
fix(ci): tell the author's agent how to triage review-bot findings - #3138
Conversation
External contributors' coding agents treat every inline finding from the review bots as a task. The result is 30-40 commit PRs that fix surrounding code and drift from their purpose (#3091 grew from 3 files to 14 on 80 bot comments and was closed as superseded by a 3-file fix). The reader of these comments is almost always the author's agent, so the comments now carry the triage rule. Every inline finding ends with a severity-aware footer: critical and major say a maintainer will expect the fix if it is a defect in code this PR changes; minor says it is the author's call. Both say to reply instead of patching anything about surrounding code and never to widen the PR. The summary opens with the same rule as a table addressed to "a coding agent acting for the author". Deep Review gets a matching footer, and AGENTS.md gets the full rule so agents load it from the contributor's checkout before they read the PR. Nothing gates merge mechanically; the text says what a maintainer will expect rather than adding a gate a hallucinated critical could trip.
deep-resolve.yml interpolates the deep review comment verbatim, so the new advisory footer lands inside the resolver's instruction block and its "P2/P3 are your call" contradicts the resolver's "Skip P3 entirely".
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🔵 Tier 2 — Low RiskSmall, isolated change with no API route or data model modifications. Why this tier:
Additional context: touches background tasks or the delivery pipeline lightly (8 lines, under the 30-line bar for Tier 4) Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns. Stats
|
Deep Review✅ No critical issues found. This PR is documentation- and CI-only (a CommonJS review helper, three workflow YAMLs,
Test coverage for the new behavior is thorough — footer presence/placement, severity mapping (including unknown→minor degradation), truncation survival, the 🔵 P3 nitpicks (1)
Reviewers (5): correctness, testing, maintainability, project-standards, previous-comments. Testing gaps: the deep-review footer gate's coupling to the review prompt's mandated finding format is not mechanically pinned; |
|
PR Review4 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 4 minor 4 posted as inline comment(s) on the changed lines. Severity is the reviewer's own estimate and is used for ordering, not filtering. |
E2E Test Results✅ All tests passed • 361 passed • 1 skipped • 1586s
Tests ran across 4 shards in parallel. |
…hell Review findings on #3138, all in this PR's own change: - AGENTS.md: "code the PR touches but did not create" overlapped with "code this PR changes", so a P0 introduced while editing an existing function fit both bullets. Use the revert test instead: did this PR introduce the problem? - deep-review.yml: the footer was a prompt-template line the model can paraphrase, drop on a clean review, or bury in the P3 block. Append it in the extract step, next to the state marker. - deep-review.yml: strip <sub> blocks from prior comments. The sibling bot's inline comments already land in prior-comments.md, and each now carried a ~250-byte footer spent from the 32KB cap ahead of human replies. - tests: the footer-precedes-marker assertion passed when the footer was missing. Add an anchor test that pins GUIDE_URL and deep-review.yml to the AGENTS.md heading, so renaming it fails CI instead of dead-linking every posted comment.
Second review round on #3138: - claude-code-review.yml: the anchor test reads AGENTS.md, but the tests run from the sparse .trusted checkout, which did not include it. After merge every review job would have failed at "Test review helpers"; it passed here only because .trusted comes from main. Add AGENTS.md to the sparse-checkout list. The test now names that list when the file is missing and asserts the list still contains it. - deep-review.yml: drop the <sub> strip. The sibling reviewer's inline comments were never excluded from prior-comments.md (mine() matched only the sticky markers), so the footer was the smaller part of what leaked, and the regex also deleted human-written <sub>. Add the per-finding dedup marker to OURS instead; human replies carry no marker and still flow through. A lockstep test pins OURS to what commentBody emits. - deep-review.yml: append the footer only when the review carries a finding and the sandbox was healthy, matching renderSummary's notice.
…oter gate on section shape Third review round on #3138: - main.yml + Makefile: run review-comments.test.mjs in ordinary CI and in make ci-unit. claude-code-review.yml runs it from a base-branch checkout, so the AGENTS.md heading guard could never fail the PR that renames the heading; it would fail every later PR's review job instead. - deep-review.yml: the footer gate matched the finding bullet's exact shape (a backtick after `- **`), which the model varies, so a format drift would silently drop the footer on every review with findings. Key it on the mandated `### ` severity heading or P3 `<details>` block instead, read $REVIEW through a here-string so an early grep -q match cannot SIGPIPE the condition under pipefail, and lift the pattern into a test that runs real grep on clean, P2, P3-only, and numbered bodies. - tests: pin deep-review.yml to the whole GUIDE_URL, not just the anchor.
…hreads (#3464) ## Summary Today the review bot posts **every** finding it can anchor as an inline thread, at every severity, labelled with its own `critical`/`major`/`minor` self-estimate. Apart from #3138 (triage guidance for the author's agent), the reviewer has not changed since #3024. This PR adds a separate grading pass that assigns each finding a priority, posts only P0–P2 inline, and folds P3 into the sticky summary. It also stops reworded findings from being re-posted as new threads on later pushes. ## Production today vs this PR | | production (`main`) | this PR | |---|---|---| | Label users see | reviewer's own severity: 🔴 critical · 🟠 major · 🔵 minor | priority only: 🔴 P0/P1 · 🟡 P2 · 🔵 P3 (same icons as deep-review) | | Inline threads | every anchored finding | P0–P2 | | Folded into the summary | only `minor` findings that could not be anchored | every P3, anchored or not. Folded, never dropped | | Grading | none | a second pass (Sonnet, `Read` only) grades the finished findings. It cannot add, drop or reword them | | Reworded finding on a later push | opens a new thread (dedup keyed on the exact title) | recognised as the same finding | | Grading fails | n/a | nothing is folded; the review posts exactly as on `main` | ### How a finding's priority is decided (`effectivePriority` in `priority.cjs`) A finding is folded only when **both** graders call it low, and nothing says it matters today: - the reviewer itself labelled it `minor`, **and** - the grading pass put it at P2 or P3, **and** - the grader doesn't say the harm is reachable in the merged code, **and** - it doesn't read as security-related. Neither grader can fold anything on its own: a grader P3 on a finding the reviewer called major or critical is shown as P2. Without a grade, nothing is folded (critical → P0, major → P1, minor → P2). ## Measured against human votes Data: 👍/👎 reactions maintainers left on this bot's inline comments, covering **75 graded findings across 29 PRs** (53 👍, 22 👎) from 4 maintainers. Two kinds of vote are excluded: reactions cast by bots, and "blanket" votes, where one person thumbed ≥10 findings on a PR without a single 👎. The "this PR" column comes from running this branch's `effectivePriority` over the grading pass's output. | | production | this PR | |---|---|---| | Inline threads | 75 / 75 | **49 / 75** | | Folded as P3 | 0 | **26 (35%)** | | 👍 major/critical findings still inline | 17 / 17 | **17 / 17** | | 👎 findings folded | 0 / 22 | 8 / 22 | | 👍 findings folded | 0 | 18, all ones the reviewer labelled `minor` | So, on this sample, about a third fewer inline threads, with no upvoted major/critical finding hidden. The cost is 18 upvoted minor findings moving from inline threads into the folded section of the summary. ## Checked against the gold set The vote data can only see the severity the bot assigned. It can't tell whether a folded finding was one a human treated as **must-fix**. That's what the gold set measures. It contains human review comments on 30 past PRs (93 items, 29 of them must-fix). The deployed reviewer's findings on those PRs were re-graded with this PR's grading prompt and routed with this PR's rule, reusing the existing match judgements. | | production | this PR | same rule, no reachable-today floor | |---|---|---|---| | Inline threads | 122 (4.1/PR) | **93 (3.1/PR)** | 70 (2.3/PR) | | Must-fix items the reviewer caught | 10/29 | 10/29 | 10/29 | | …of those, folded out of the inline threads | 0 | **0** | **3** | Must-fix recall can't move, because grading never changes what was found. The column to read is the last row. The grader rated 3 of the 10 caught must-fix items P2, and the reviewer had labelled all 3 `minor`, so the minor-P2 fold alone would have hidden them: - #1410: deep-linked equality/range filters are silently dropped from the filter sidebar - #2604: OpenAPI spec not regenerated for the new tile type - #2696: `getRecentAlertGroupSummaries` has no DB-side document cap, unlike its sibling The reachable-today floor is what keeps all three inline. On this data it is the difference between hiding 0 and 3 of the 10. ## Dedup fix The cross-push dedup key was `file + exact title`, on the assumption that titles are stable between runs. They aren't. #3039 was reviewed four times in 2.3 hours, and one defect was posted three times under three titles; all three were thumbed down. Restatements are now matched by title-word overlap within the same file: at least 0.4 overlap **and** at least 3 shared content words. The second condition stops short titles like "Missing timeout in fetchUser" and "Missing retry in fetchUser" from merging. The threshold (0.4) was set from all 140 same-PR, same-file pairs of voted findings. Every pair scoring 0.36 or higher is a restatement. At 0.333, restatements collide with genuinely distinct findings. On #3129, "signs every user out on each container restart" and "still signs sessions with the public secret" are two real defects (the second critical) at exactly that score. So one known reword on #3039 is still missed rather than risk merging those two. Both cases are verbatim test fixtures. ## Limits and what isn't measured yet - **The gold-set result rests on 10 caught must-fix items** from one reviewer run (Sonnet; production runs Opus). Zero folded is the bar, but each item is worth 10 points. - **The votes are sparse and uneven.** There are 22 👎, mostly from two people whose thresholds differ ~4×, and only 1 of 197 voted comments has more than one reactor. - **The reachable-today floor costs volume.** In the vote sample it didn't separate 👍 from 👎 (~30% 👎 either way), and it lowers the fold rate there from 52% to 35%. On the gold set it is what keeps all 3 at-risk must-fix items visible, so the cost buys something. - **Premise findings.** The grading prompt describes findings that object to the change's premise as "published in their own section". This PR does not build that section; those findings are routed by priority like any other. - **Cost:** one more Sonnet call per reviewed push that has at least one finding. Not yet measured in CI. - **This PR's own review runs the old workflow,** because `pull_request_target` loads it from `main`. To exercise the new path before merge, dispatch **Claude Code Review** from this branch against an open PR. ## Test plan - [x] `node --test .github/scripts/code-review/__tests__/*.test.mjs`: 58/58. The priority rule and the title dedup live in `priority.cjs` and `dedup.cjs`. - [x] Workflow YAML parses; Prettier and knip clean on the changed files - [x] Ran the helpers over a real PR's findings and grades (#3258: 7 findings → 4 shown, 3 folded) - [ ] Dispatch from this branch on 2–3 open PRs and check the inline/folded split - [x] Gold-set check: 0 of 10 caught must-fix items folded (3 without the reachable-today floor) ## Also tried (not in this PR) - **The reviewer assigning P0–P3 itself, in a single pass.** Its grades were better calibrated, but it caught 7/29 must-fix items against 10/29 (one run each, so possibly noise). - **Folding all P2s, not just minor ones.** This hid 61% of comments but lost 4 of the 17 upvoted major/critical findings. - **Folding on the reviewer's `minor` label alone.** On the gold set it hid 46% of the findings humans treated as merge-blocking. - **Using `reachableToday` as the main signal.** It didn't distinguish 👍 from 👎 in the votes, so here it only acts as a floor. As a floor, it's what protects must-fix items on the gold set. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Summary
Review bots post findings. The author's coding agent reads them as a task list and fixes all of them, nits and "hoist this helper so another file can use it" suggestions included, which is how #3091 grew from 3 files to 14 before it was closed as superseded. Every inline comment now ends with a short footer keyed to severity, and the sticky summary opens with the same rule, so the reader knows what a maintainer expects of a critical finding versus a nit, and what to do with a finding about code the PR only sits next to. Deep Review's summary gets the same footer, and both link to a new "Responding to review feedback" section in
AGENTS.md.deep-resolve.ymlnow says its own rules beat the review body it interpolates verbatim, so the footer cannot loosen/just-fix-it's "skip P3 entirely".How to test on Vercel preview
N/A. Non-UI change (CI workflows and repo docs only).
References