Repository navigation
fix(mcp): time-limit the trace top-operations tool and explain timeouts - #3258
Conversation
The clickstack_trace_waterfall tool ran TraceId-only span/log queries that scanned the full retention window and timed out. Thread the search window into both queries, add a max_execution_time ceiling, and probe the trace's actual [min, max] span extent so an explicit traceId older than the default window — or one that ran longer than an hour — still resolves in full. The probe also runs for auto-picked traces, so a picked trace whose root predates the window is no longer truncated into a partial tree. The fetch window is width-clamped to the recent tail so a reused/sentinel traceId can't widen the scan back toward the retention edge or stitch unrelated occurrences into one tree. Empty-result hints name the recoverable action (pass an explicit startTime), and probe failures are surfaced rather than silently swallowed. ClickHouse query timeouts are reclassified from user to server errors.
- MCP ClickHouse settings now win over source.querySettings so a source can't relax the max_execution_time / readonly ceiling (span + log fetches). - Surface a probeNote when the extent probe fails and the fallback window may return a partial tree, instead of presenting it as complete. - Surface a windowNote when the fetch-window cap drops a wide-spread trace's earliest spans, so the tool no longer silently promises the whole tree. - Drop the newly added "Safety limits" ASCII divider per AGENTS.md.
- errorHint: match only real timeouts (TIMEOUT_EXCEEDED / "Timeout exceeded") so a "Setting max_execution_time shouldn't be greater than…" constraint error no longer hijacks the timeout hint. Regression test added. - Wrap tsExpr / logTsExpr in getFirstTimestampValueExpression for every raw-SQL use (probe min/max, tree + log SELECT/WHERE/ORDER BY, picker orderBy) so a multi-column timestampValueExpression (e.g. "EventDate, EventTime") no longer breaks the probe and span queries. - Probe in auto-pick mode even with an explicit startTime: startTime is the pick window there, not a fetch bound, so the picked trace's full extent must still be probed. Only explicit traceId + explicit startTime honors the window verbatim. Integration test added. - Give LONG_TRACE_ID a unique value so it no longer collides with the breakdown suite's TRACE_ID_1 in the never-cleared traces table.
clickstack_trace_top_time_consuming_operations built its ClickHouse client with the 1h default request timeout and set no max_execution_time, so on self-hosted it could run for an hour; on ClickHouse Cloud a third of calls ended in a bare "Timeout error.". - Use MCP_REQUEST_TIMEOUT and MCP_CLICKHOUSE_SETTINGS like the other MCP query tools. MCP settings override source.querySettings so a source can't lift the time limit or the readonly guard. - On timeout, tell the agent to shorten the window, scope the filter or set minParentDurationMs instead of saying the SQL is invalid. - Recognise the client-side "Timeout error." in errorHint. - Use the first timestamp expression so multi-column sources build valid SQL. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: 13d8ca1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
PR Review
2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor 1 posted as inline comment(s) on the changed lines. 1 listed below. Findings outside the changed lines1 minor
Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
|
Deep Review✅ No critical issues found. No P0/P1 defects: correctness, security, performance, project-standards, and maintainability reviewers all came back clean, and the refactor is well-contained. The settings-precedence change is a net security improvement (a source's 🟡 P2 -- recommended
🔵 P3 nitpicks (5)
Reviewers (8): correctness, security, performance, project-standards, testing, kieran-typescript, maintainability, previous-comments. Testing gaps:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
Date-first multi-column timestampValueExpression (e.g. EventDate, EventTime) compared a Date against millisecond bounds and dropped every span after midnight. Render the time filters with timeFilterExpr (greptile P1), cover the multi-column path with an integration test (deep-review P2), and add the missing changeset.
…s (deep-review P3)
…akdown-timeout # Conflicts: # packages/api/src/mcp/__tests__/query.test.ts # packages/api/src/mcp/__tests__/trace.int.test.ts # packages/api/src/mcp/tools/query/helpers.ts # packages/api/src/mcp/tools/trace/waterfall.ts
🟡 Tier 3 — StandardIntroduces new logic, modifies core functionality, or touches areas with non-trivial risk. Why this tier:
Review process: Full human review — logic, architecture, edge cases. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
E2E Test Results✅ All tests passed • 395 passed • 1 skipped • 1318s
Tests ran across 4 shards in parallel. |
Drops the duplicate regex-based helper. The existing one also catches socket ETIMEDOUT, aborts, and ClickHouseError code 159, so those no longer get the invalid-SQL suffix (inline review minor).
- Move getMcpClickhouseClient, mcpQuerySettings, durationDivisor and the inclusive timeFilterExpr wrapper into trace/shared.ts; waterfall and breakdown both use them, so the MCP-settings-last order lives in one place. - Move the breakdown schema and tool description to breakdownSchema.ts (breakdown.ts 400 -> 266 lines). - errorHint also checks isQueryTimeoutError on the error object, so socket timeouts and aborts get the execution-time hint. - Type breakdown operations in tests instead of any.
- isQueryOutOfTime excludes socket ETIMEDOUT (ClickHouse unreachable) from the execution-time hint and the breakdown timeout suffix (greptile P2). - Connection/server errors in the breakdown query no longer get the invalid-SQL suffix either. - Metadata-lookup failures drop the 'shorten the window' suffix; those DESCRIBE queries don't depend on the window or filters (inline minor). - Restore main's timeout regex; the 'Timeout error.' alternative was dead since errorHint always gets the error object (inline minor).
…hreads (#3464) ## Summary Today the review bot posts **every** finding it can anchor as an inline thread, at every severity, labelled with its own `critical`/`major`/`minor` self-estimate. Apart from #3138 (triage guidance for the author's agent), the reviewer has not changed since #3024. This PR adds a separate grading pass that assigns each finding a priority, posts only P0–P2 inline, and folds P3 into the sticky summary. It also stops reworded findings from being re-posted as new threads on later pushes. ## Production today vs this PR | | production (`main`) | this PR | |---|---|---| | Label users see | reviewer's own severity: 🔴 critical · 🟠 major · 🔵 minor | priority only: 🔴 P0/P1 · 🟡 P2 · 🔵 P3 (same icons as deep-review) | | Inline threads | every anchored finding | P0–P2 | | Folded into the summary | only `minor` findings that could not be anchored | every P3, anchored or not. Folded, never dropped | | Grading | none | a second pass (Sonnet, `Read` only) grades the finished findings. It cannot add, drop or reword them | | Reworded finding on a later push | opens a new thread (dedup keyed on the exact title) | recognised as the same finding | | Grading fails | n/a | nothing is folded; the review posts exactly as on `main` | ### How a finding's priority is decided (`effectivePriority` in `priority.cjs`) A finding is folded only when **both** graders call it low, and nothing says it matters today: - the reviewer itself labelled it `minor`, **and** - the grading pass put it at P2 or P3, **and** - the grader doesn't say the harm is reachable in the merged code, **and** - it doesn't read as security-related. Neither grader can fold anything on its own: a grader P3 on a finding the reviewer called major or critical is shown as P2. Without a grade, nothing is folded (critical → P0, major → P1, minor → P2). ## Measured against human votes Data: 👍/👎 reactions maintainers left on this bot's inline comments, covering **75 graded findings across 29 PRs** (53 👍, 22 👎) from 4 maintainers. Two kinds of vote are excluded: reactions cast by bots, and "blanket" votes, where one person thumbed ≥10 findings on a PR without a single 👎. The "this PR" column comes from running this branch's `effectivePriority` over the grading pass's output. | | production | this PR | |---|---|---| | Inline threads | 75 / 75 | **49 / 75** | | Folded as P3 | 0 | **26 (35%)** | | 👍 major/critical findings still inline | 17 / 17 | **17 / 17** | | 👎 findings folded | 0 / 22 | 8 / 22 | | 👍 findings folded | 0 | 18, all ones the reviewer labelled `minor` | So, on this sample, about a third fewer inline threads, with no upvoted major/critical finding hidden. The cost is 18 upvoted minor findings moving from inline threads into the folded section of the summary. ## Checked against the gold set The vote data can only see the severity the bot assigned. It can't tell whether a folded finding was one a human treated as **must-fix**. That's what the gold set measures. It contains human review comments on 30 past PRs (93 items, 29 of them must-fix). The deployed reviewer's findings on those PRs were re-graded with this PR's grading prompt and routed with this PR's rule, reusing the existing match judgements. | | production | this PR | same rule, no reachable-today floor | |---|---|---|---| | Inline threads | 122 (4.1/PR) | **93 (3.1/PR)** | 70 (2.3/PR) | | Must-fix items the reviewer caught | 10/29 | 10/29 | 10/29 | | …of those, folded out of the inline threads | 0 | **0** | **3** | Must-fix recall can't move, because grading never changes what was found. The column to read is the last row. The grader rated 3 of the 10 caught must-fix items P2, and the reviewer had labelled all 3 `minor`, so the minor-P2 fold alone would have hidden them: - #1410: deep-linked equality/range filters are silently dropped from the filter sidebar - #2604: OpenAPI spec not regenerated for the new tile type - #2696: `getRecentAlertGroupSummaries` has no DB-side document cap, unlike its sibling The reachable-today floor is what keeps all three inline. On this data it is the difference between hiding 0 and 3 of the 10. ## Dedup fix The cross-push dedup key was `file + exact title`, on the assumption that titles are stable between runs. They aren't. #3039 was reviewed four times in 2.3 hours, and one defect was posted three times under three titles; all three were thumbed down. Restatements are now matched by title-word overlap within the same file: at least 0.4 overlap **and** at least 3 shared content words. The second condition stops short titles like "Missing timeout in fetchUser" and "Missing retry in fetchUser" from merging. The threshold (0.4) was set from all 140 same-PR, same-file pairs of voted findings. Every pair scoring 0.36 or higher is a restatement. At 0.333, restatements collide with genuinely distinct findings. On #3129, "signs every user out on each container restart" and "still signs sessions with the public secret" are two real defects (the second critical) at exactly that score. So one known reword on #3039 is still missed rather than risk merging those two. Both cases are verbatim test fixtures. ## Limits and what isn't measured yet - **The gold-set result rests on 10 caught must-fix items** from one reviewer run (Sonnet; production runs Opus). Zero folded is the bar, but each item is worth 10 points. - **The votes are sparse and uneven.** There are 22 👎, mostly from two people whose thresholds differ ~4×, and only 1 of 197 voted comments has more than one reactor. - **The reachable-today floor costs volume.** In the vote sample it didn't separate 👍 from 👎 (~30% 👎 either way), and it lowers the fold rate there from 52% to 35%. On the gold set it is what keeps all 3 at-risk must-fix items visible, so the cost buys something. - **Premise findings.** The grading prompt describes findings that object to the change's premise as "published in their own section". This PR does not build that section; those findings are routed by priority like any other. - **Cost:** one more Sonnet call per reviewed push that has at least one finding. Not yet measured in CI. - **This PR's own review runs the old workflow,** because `pull_request_target` loads it from `main`. To exercise the new path before merge, dispatch **Claude Code Review** from this branch against an open PR. ## Test plan - [x] `node --test .github/scripts/code-review/__tests__/*.test.mjs`: 58/58. The priority rule and the title dedup live in `priority.cjs` and `dedup.cjs`. - [x] Workflow YAML parses; Prettier and knip clean on the changed files - [x] Ran the helpers over a real PR's findings and grades (#3258: 7 findings → 4 shown, 3 folded) - [ ] Dispatch from this branch on 2–3 open PRs and check the inline/folded split - [x] Gold-set check: 0 of 10 caught must-fix items folded (3 without the reachable-today floor) ## Also tried (not in this PR) - **The reviewer assigning P0–P3 itself, in a single pass.** Its grades were better calibrated, but it caught 7/29 must-fix items against 10/29 (one run each, so possibly noise). - **Folding all P2s, not just minor ones.** This hid 61% of comments but lost 4 of the 17 upvoted major/critical findings. - **Folding on the reviewer's `minor` label alone.** On the gold set it hid 46% of the findings humans treated as merge-blocking. - **Using `reachableToday` as the main signal.** It didn't distinguish 👍 from 👎 in the votes, so here it only acts as a floor. As a floor, it's what protects must-fix items on the gold set. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Follow-up to #3116, which gave
clickstack_trace_waterfallthe MCP timeouts. This PR does the same forclickstack_trace_top_time_consuming_operations.Why
The tool built its ClickHouse client with the 1h default request timeout and set no
max_execution_time, so on self-hosted it could run for an hour. When it did time out, the agent got a bare{"error":"Timeout error."}with a suffix saying the parentFilter SQL was invalid.What changed
breakdown.tsusesMCP_REQUEST_TIMEOUT(32s) andMCP_CLICKHOUSE_SETTINGS(max_execution_time: 30,readonly: 2), matching the other MCP query tools.source.querySettingscan no longer overridemax_execution_timeorreadonly. MCP settings are spread last, the same as the waterfall tool. Before this change, querySettings were spread afterreadonly: '1', so a source could override the readonly guard too.readonlymoves from 1 to 2 because readonly=1 rejects settingmax_execution_time. readonly=2 still blocks DDL/DML injected throughparentFilter.TIMEOUT_EXCEEDEDor the client'sTimeout error.), the error now tells the agent to shorten the window, scopeparentFilterto one ServiceName + SpanName, setminParentDurationMs, or lowermaxParentTraces. Other errors keep the "invalid SQL" suffix.helpers.ts: newisQueryTimeoutError().errorHintnow matches the client-sideTimeout error., so every MCP tool shows the timeout hint for it, not just this one.timeFilterExpr, which filters on every column of a multi-columntimestampValueExpressionand compares Date columns at day precision. Multi-column sources (e.g.TimestampTime, Timestamp) previously produced invalid SQL, and taking only the first column would drop every span after midnight for a date-first source likeEventDate, EventTime.Why the query is slow (not fixed here)
parentFilternames ServiceName and SpanName, because those lead theORDER BY (ServiceName, SpanName, toDateTime(Timestamp))key.TraceId IN (parent_traces)) can't use the sort key. The TraceId skip index can't prune many granules when the set holds up to 100k IDs (maxParentTracesdefault), and most granules hold a span from some matching trace anyway. So stage 2 reads TraceId for every span in the window ±60s. Cost tracks window length × ingest rate, not how many parents matched.trace_id_tsMV, which isn't available on every schema; bound stage 2 to the parents' actual min/max time span; or lower themaxParentTracesdefault and sample. For now the timeout message points agents at the window length, the lever that matters.Tests
query.test.ts):isQueryTimeoutErrormatches client/server timeouts and wrapped causes, and does not match constraint-violation errors.errorHintmatchesTimeout error..trace.int.test.ts): checks that the breakdown query sendsmax_execution_time: 30/readonly: 2even when the source setsmax_execution_time: 3600, that other querySettings still pass through, and that the client usesMCP_REQUEST_TIMEOUT. Also checks that both timeout shapes return the actionable message and not the invalid-SQL suffix. 37/37 pass.🤖 Generated with Claude Code