Repository navigation
fix: make "Denoise results" remove noise reliably - #3621
Conversation
- Mine the source's body column instead of the last table column. Adding an attribute column from the side panel switched denoise to that column, or broke the sample query when the column carried an alias. - Strip ANSI codes before matching, as sampling already did, so colored rows of a noisy pattern are removed. - Replace drain3's match() with the template-fit matcher the MCP denoise uses, moved to common-utils and shared by both. - Say in the panel when results can't be denoised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: 45de3c6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🟡 Tier 3 — StandardIntroduces new logic, modifies core functionality, or touches areas with non-trivial risk. Why this tier:
Review process: Full human review — logic, architecture, edge cases. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
PR Review
8 finding(s): 🔴 0 critical · 🟠 2 major · 🔵 6 minor 6 posted as inline comment(s) on the changed lines. 2 listed below. Findings outside the changed lines
1 minor
Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
Deep ReviewScope: PR #3621 — "Denoise results" reliability fix across The refactor is clean: the old ✅ No critical issues found. 🟡 P2 — recommended
🔵 P3 nitpicks (4)
Reviewers (6): correctness, adversarial, kieran-typescript, testing, maintainability, performance. Testing gaps: No test covers a level-less source where the error/warning guarantee degrades; no benchmark guards the synchronous denoise matcher at live-tail scale; Pre-existing (not counted): The app (pyodide drain3 with No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
E2E Test Results✅ All tests passed • 464 passed • 1 skipped • 1330s
Tests ran across 4 shards in parallel. |
Drain splits one logical pattern across clusters by token count and first token, so on the demo logs no cluster passed 10% and denoise removed nothing. Take the most frequent patterns until they cover half the sample, ignoring patterns under 1%. Applies to the web app and the MCP denoise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Never hide a pattern whose sampled lines include an error or warning,
so a flood of errors during an incident stays visible.
- Replace the raw template list with a one-line summary ("Hiding N
common patterns, about X% of events") and a collapsed list. Each
pattern is shortened to one line, with runs of wildcards collapsed to
an ellipsis and its share of events.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fixed cutoff is easier to reason about than "the top patterns up to half the sample". 5% still catches the dominant demo patterns that sat below the old 10% because Drain splits them across clusters. Error and warning patterns stay excluded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #3544 (option 1) Option 2 (comparing both windows in one Drain tree) moved to #3556 / Draft #3557, stacked on this PR. Option 3 (`bodyExpression`) is left for a new issue after #3426, as agreed on #3544. Merged with `main` after #3426 landed: the drill-down uses #3426's predicate. Merged again after #3621, which moved `buildNoisyBodyMatcher` into common-utils and shares it with the app (see 3). ## What 1. **Default masking before Drain** (`packages/common-utils/src/drain/config.ts`). `TemplateMinerConfig` starts with `DEFAULT_MASKING_INSTRUCTIONS`, which mask only the date of a timestamp to `<DATE>` (exported as `DATE_MASK`): ISO dates (also before `T` in ISO-8601), RFC-1123, ctime and Java `Date.toString`, JavaScript `Date.toString`, `java.util.logging`, syslog, access log, Tomcat, `dd/MM/yyyy`, `MM/dd/yyyy`, `dd.MM.yyyy`, Spark `yy/MM/dd` and the klog `I1104` date. Forms other than ISO dates are masked only before a time. Times and ids are not masked. Every line of a day shares its date, and `<DATE>` matches itself the same way, so within one day lines group as without masks (tested); across days the templates are the same. One change within a day: klog lines with the same message but different severity letters are no longer merged, because `I<DATE>` and `E<DATE>` route by their letter. No mask changes a line's token count: month-name dates are masked word by word through a new optional `perWord` flag on masking instructions (default off, as in drain3). Every caller that builds a `TemplateMinerConfig` (pattern mining, denoise, the CLI) gets the masks; `TemplateMinerConfig.fromJSON` applies none unless the JSON lists them, as drain3 does. 2. **Drill-down** (`templateToRegex` in `patternDrilldown.ts`): `<DATE>` matches one or more non-whitespace characters, also inside a token (`backup_<DATE>.tar`). A whole-token `<*>` is unchanged, so #3426's matching rules and tests stay. 3. **Denoise** (`buildNoisyBodyMatcher`): with `masked: true`, masks each body the same way before comparing it with the noisy templates. The MCP denoise passes it; the app's Denoise mines with drain3's default config, which has no masks, so it keeps comparing unmasked bodies. The web app's Event patterns view runs drain3 in Pyodide with its own config and is unchanged. ## Why On dated logs (Java/Logback, Python, Airflow run ids), `emerging_signals` with a yesterday baseline reported the same pattern as both emerging and disappeared, because the templates differed only by the date (#3544: 19–135 false patterns per run on 4 services). ## Tests - `drain-masking.test.ts`: each mask on real-looking lines; lines left alone (versions, hex, short shas, times, ids, `5 Mar`, `10.11.2026` without a time, `1234-56-78`); a mixed day of logs clusters the same with and without masks (klog has its own test); two days and a month boundary give the same templates; word counts never change; `extractParameters` reads a per-word date back; drain3 `fromJSON` behaviour; linear time on a 128 KiB whitespace run. - `patternDrilldown.test.ts`: `<DATE>` as a word and inside a token, mined dated lines whose predicate matches every own sample, and a dated line next to `Started job at boot`. - `noisy-matcher.test.ts`: with `masked: true`, dated noisy lines are removed and a rare line is kept; without it, a template with a literal date (as drain3 keeps one shared by every line) still matches. - 3,090 common-utils tests, 560 MCP unit tests and the emerging_signals and query tool integration tests (116) pass; tsc, eslint and prettier clean. Co-authored-by: Drew Davis <6097246+pulpdrew@users.noreply.github.com>
Summary
"Denoise results" on the search page often left noisy rows in the table. Three bugs caused it:
patternColumn = columns[columns.length - 1]. Adding a column from the side panel appends it, so denoise switched to that attribute. An added map column carries an alias (LogAttributes['k'] AS "k"), and the alias was then used as a SQL expression in the sample query. Denoise now uses the source's body expression (getEventBody), finds its result column by position in the select list, and says so in the panel when the body isn't in the table.match(). That needs an exact template fit and can put a row in a different cluster from the one it was counted in. The MCP denoise already replaced this withbuildNoisyBodyMatcher. That matcher moves tocommon-utils/src/drain/noisy-matcher.ts, gains anextraDelimitersoption so it tokenizes like the web miner, and is now used by both.The denoise logic moves out of
DBRowTable.tsxintohooks/useDenoisedRows.tsandcomponents/DenoiseSummary.tsx. If the pattern query fails, the table now shows all rows instead of an empty table.Noise threshold: 10% → 5%. A pattern counts as noisy when its cluster holds more than the threshold share of the 10k-row sample. Drain splits one logical pattern across clusters by token count and first token, so on the demo logs no cluster passed 10% (the top one was 9.2%) and denoise removed nothing. At 5%, the demo data's 6-hour window hides 6–7 patterns (the sample is random) covering about 45% of it. The threshold lives in
selectNoisyPatterns(common-utils), shared by the app and the MCPdenoiseoption, whose tool description is updated. The MCP integration test still passes.Errors and warnings are never hidden. Frequency isn't importance: during an incident a flood of errors becomes the most common pattern. The app skips any pattern whose sampled lines include an error or warning (
getLogLevelClass). The MCP denoise doesn't apply this yet, because its sample query doesn't select the level.Readable panel. The raw template list is replaced by a one-line summary ("Hiding 6 common patterns, about 45% of events. Errors and warnings are never hidden.") with a "Show patterns" toggle. Each pattern is one truncated line with its share of events, and runs of wildcards collapse to
…(simplifyPattern). Hover shows the full template.Screenshots or video
Demo Logs on the Vercel previews (
mainand this branch), same 6-hour window, "Denoise results" on.How to reproduce
Starting point:
yarn app:dev:localconnected to the demo server (onboarding → "Connect to Demo Server").On
main:On this branch:
Bodycolumn from the select. The panel reads "Add Body to the table columns to denoise results." and all rows are shown.How to test on Vercel preview
Preview routes: /search
Steps:
Timestamp, ServiceNameand run the search.References
buildNoisyBodyMatcher)🤖 Generated with Claude Code