Skip to content

fix(patterns): mask dates before Drain - #3551

Merged
kodiakhq[bot] merged 15 commits into
ClickHouse:mainfrom
bcanvural:fix/patterns-mask-dated-lines
Nov 19, 2026
Merged

kodiakhq[bot] merged 15 commits into
ClickHouse:mainfrom
bcanvural:fix/patterns-mask-dated-lines

Conversation

@bcanvural

@bcanvural bcanvural commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Refs #3544 (option 1)

Option 2 (comparing both windows in one Drain tree) moved to #3556 / Draft #3557, stacked on this PR. Option 3 (bodyExpression) is left for a new issue after #3426, as agreed on #3544. Merged with main after #3426 landed: the drill-down uses #3426's predicate. Merged again after #3621, which moved buildNoisyBodyMatcher into common-utils and shares it with the app (see 3).

What

  1. Default masking before Drain (packages/common-utils/src/drain/config.ts). TemplateMinerConfig starts with DEFAULT_MASKING_INSTRUCTIONS, which mask only the date of a timestamp to <DATE> (exported as DATE_MASK): ISO dates (also before T in ISO-8601), RFC-1123, ctime and Java Date.toString, JavaScript Date.toString, java.util.logging, syslog, access log, Tomcat, dd/MM/yyyy, MM/dd/yyyy, dd.MM.yyyy, Spark yy/MM/dd and the klog I1104 date. Forms other than ISO dates are masked only before a time.

    Times and ids are not masked. Every line of a day shares its date, and <DATE> matches itself the same way, so within one day lines group as without masks (tested); across days the templates are the same. One change within a day: klog lines with the same message but different severity letters are no longer merged, because I<DATE> and E<DATE> route by their letter.

    No mask changes a line's token count: month-name dates are masked word by word through a new optional perWord flag on masking instructions (default off, as in drain3). Every caller that builds a TemplateMinerConfig (pattern mining, denoise, the CLI) gets the masks; TemplateMinerConfig.fromJSON applies none unless the JSON lists them, as drain3 does.

  2. Drill-down (templateToRegex in patternDrilldown.ts): <DATE> matches one or more non-whitespace characters, also inside a token (backup_<DATE>.tar). A whole-token <*> is unchanged, so fix(mcp): make event_patterns drill-down snippets executable #3426's matching rules and tests stay.

  3. Denoise (buildNoisyBodyMatcher): with masked: true, masks each body the same way before comparing it with the noisy templates. The MCP denoise passes it; the app's Denoise mines with drain3's default config, which has no masks, so it keeps comparing unmasked bodies.

The web app's Event patterns view runs drain3 in Pyodide with its own config and is unchanged.

Why

On dated logs (Java/Logback, Python, Airflow run ids), emerging_signals with a yesterday baseline reported the same pattern as both emerging and disappeared, because the templates differed only by the date (#3544: 19–135 false patterns per run on 4 services).

Tests

  • drain-masking.test.ts: each mask on real-looking lines; lines left alone (versions, hex, short shas, times, ids, 5 Mar, 10.11.2026 without a time, 1234-56-78); a mixed day of logs clusters the same with and without masks (klog has its own test); two days and a month boundary give the same templates; word counts never change; extractParameters reads a per-word date back; drain3 fromJSON behaviour; linear time on a 128 KiB whitespace run.
  • patternDrilldown.test.ts: <DATE> as a word and inside a token, mined dated lines whose predicate matches every own sample, and a dated line next to Started job at boot.
  • noisy-matcher.test.ts: with masked: true, dated noisy lines are removed and a rare line is kept; without it, a template with a literal date (as drain3 keeps one shared by every line) still matches.
  • 3,090 common-utils tests, 560 MCP unit tests and the emerging_signals and query tool integration tests (116) pass; tsc, eslint and prettier clean.

…gnals windows in one tree

Dated log lines kept their literal date in the template, and emerging_signals
mined the baseline and current windows in separate trees and matched them by
template text, so the same line showed up as both new and gone. Mask
timestamps (ISO-8601/RFC-3339, RFC-1123, access-log, syslog), UUIDs and long
hex ids by default, mine both windows with one TemplateMiner and compare by
cluster id. End the baseline 1 ms early so a row at the shared boundary isn't
counted in both windows. Refs ClickHouse#3544.
@bcanvural
bcanvural requested a review from a team as a code owner November 4, 2026 16:03
@bcanvural
bcanvural requested a review from pulpdrew November 4, 2026 16:03
@changeset-bot

changeset-bot Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: a1098fd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/cli Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Nov 4, 2026

Copy link
Copy Markdown

@bcanvural is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 2 P3

No inline comments. 2 P3 folded at the bottom.

🔵 2 P3 — no action needed before merge
  • 🔵 P3 packages/api/src/mcp/tools/query/runEventPatterns.ts:475 — Callers must restate by hand which masks minePatterns applied (masks: [DATE_MASK] here, masked: true in denoise.ts) → Three things now have to agree, and nothing ties them together: the masks in DEFAULT_MASKING_INSTRUCTIONS (config.ts), the hardcoded [DATE_MASK] passed here, and the masked: true flag in denoise.ts:232 (which goes through createPatternMasker, a second new TemplateMinerConfig() copied from mine-patterns.ts:151). Suppose someone later adds a mask with a new name, such as <IP>. minePatterns would start emitting <IP> in templates, but this predicate would still treat <IP> as literal text, so whereSnippet silently matches zero rows. The two options also have different shapes for the same idea (masks: string[] and masked: boolean). Fix: export the mask names from the single config, e.g. PATTERN_MASK_NAMES = LogMasker.fromConfig(new TemplateMinerConfig()).maskNames, or have minePatterns return its masker or mask names. Use that here and in buildNoisyBodyMatcher instead of literals.
  • 🔵 P3 packages/common-utils/src/drain/config.ts:84 — The masks are added to the drain3 port's own default config, so drain3-faithful users now have to opt out → TemplateMinerConfig is a drain3 port, and drain3 applies no masks by default. Turning the masks on in the class default means fromJSON now has to clear them (line 93), and tests that want drain3 behaviour have to clear them too (pattern-predicate.test.ts mineClusters). Any future new TemplateMinerConfig() meant to mirror the app's drain3 miner will quietly mask dates. Fix: keep the class default at [], and set maskingInstructions = [...DEFAULT_MASKING_INSTRUCTIONS] only in the callers that want it: minePatterns (mine-patterns.ts:151), createPatternMasker, and the CLI --patterns path (cli.tsx:2376).

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a focused, well-tested change. Date masking is correctly gated behind the masks option so only the MCP drill-down opts in; injection and ReDoS surfaces were checked and are sound (the ctime-year lookbehind is gated by a cheap leading lookahead, keeping it linear, and every token still flows through escapeRe2Literal → escapeSqlString); the token-count invariant holds across all masked forms including multi-space ctime; and all prior review feedback (the masks gate, compactRegex parity, test relocation, DATE_TOKEN export) is addressed. The findings below are maintainability and test-coverage recommendations only.

🟡 P2 -- recommended

  • packages/common-utils/src/drain/mine-patterns.ts:26 -- createPatternMasker() rebuilds new TemplateMinerConfig() to mask denoise bodies "the way minePatterns does", but minePatterns independently constructs its own config at line 151, so the mask-set equality that buildNoisyBodyMatcher({ masked: true }) relies on is enforced only by a comment; if minePatterns ever accepts a caller-supplied config, the denoise masker silently tokenizes bodies differently than the templates were mined with.
    • Fix: Have both paths obtain masks from one shared config factory, or pass the config minePatterns actually used into createPatternMasker.
    • maintainability
🔵 P3 nitpicks (4)
  • packages/common-utils/src/drain/pattern-predicate.ts:90 -- tokenRegex recurses over masks.slice(1) to support an arbitrary mask list, but every production and test caller passes exactly [DATE_MASK], so the multi-mask branch is both speculative generality and the only untested path in the function.
    • Fix: Collapse to a single-mask split, or add a two-mask test that pins the recursion's contract.
    • maintainability, testing, kieran-typescript
  • packages/common-utils/src/drain/config.ts:44 -- The ordering dependency in DEFAULT_MASKING_INSTRUCTIONS is load-bearing (the ctime-year instruction's lookbehind must run before the month/day instruction rewrites those tokens) but is noted only on the one instruction, so the array reads as reorderable.
    • Fix: Add a header comment stating the array order is significant and add a focused ctime test (Nov 4 06:12:12 UTC 2026) to lock the dependency.
    • maintainability
  • .changeset/patterns-mask-dated-lines.md:1 -- new TemplateMinerConfig() now date-masks by default (a behavior change to the default mining output), yet the bump is patch; defensible because @hyperdx/common-utils is private and @hyperdx/cli adopts the new default intentionally, but worth a maintainer confirmation.
    • Fix: Confirm patch is intended, or bump @hyperdx/common-utils to minor to signal the default-behavior change.
    • api-contract, project-standards, previous-comments
  • packages/common-utils/src/drain/config.ts:119 -- Pre-existing: the masking_instructions parser asserts item is Record<string, string> from only an object/null check, so pattern/maskWith can resolve to undefined and later reach new RegExp(undefined, 'g'); this diff adds the maskingInstructions = [] reset directly above it but does not introduce the unsoundness, and fromJSON has no production callers.
    • Fix: Narrow to Record<string, unknown> and drop entries whose resolved pattern/maskWith are not strings.
    • kieran-typescript

Reviewers (9): security, testing, maintainability, performance, api-contract, project-standards, previous-comments, kieran-typescript, learnings-researcher.

Testing gaps:

  • parsePatternPredicate round-trip is documented for mask-bearing templates (whole-token <DATE> reads back as <*>; in-token masks parse to null) but is untested with the masks option; note no production caller parses mask-bearing predicates today (the app's parsePatternFilter passes no masks), which lowers the risk.
  • No head-to-head denoise test asserts masked: true is load-bearing — i.e. the same <DATE>-bearing body is removed only with the flag and kept without it.
  • The linear-time guard asserts an absolute wall-clock bound (< 1000ms), which can flake under CI contention and masks sub-catastrophic regressions; comparing scaling across two input sizes would test linearity more robustly.

Coverage note: The correctness, adversarial, reliability, and agent-native reviewers did not return before finalization. The highest-stakes dimensions they target were independently verified during synthesis (token-count invariance across all masked forms, tokenRegex recursion and whole-token/in-token behavior, masking-instruction ordering, createPatternMasker/minePatterns config equivalence) and by the security reviewer (injection and ReDoS), none of which surfaced a P0/P1 issue.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

…pen baseline, drop masked tokens from whereSnippet

Fetch both windows in parallel, then mine baseline and current into the shared
miner in that order, so the result doesn't depend on which query finishes
first; report each cluster's template from the finished tree. End the baseline
with an exclusive bound instead of 1 ms early. Drop tokens that contain a mask
from whereSnippet. Narrow the changeset to the MCP/CLI paths (the web app's
Event patterns view uses drain3 with its own config).
@bcanvural

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews. Addressed in 8c44344:

  • 🟠 whereSnippet with masks inside a token ([<*>], run_id=scheduled__<*>, trace_id=<*>): the builder now drops any token that contains <*>, not only bare <*> tokens (extracted as buildWhereSnippet, with a test for a bracketed timestamp and key=id tokens). Kept minimal, since fix(mcp): make event_patterns drill-down snippets executable #3426 reworks this builder.
  • 🟠 Insertion order in the shared tree: the windows are now fetched in parallel and then mined in a fixed order, baseline first (mineWindowPatterns is split into fetchWindowSample + mineWindowSample), and each cluster's reported template is read from the finished tree. New unit test: same result whichever window query finishes first.
  • 🔵 / P2 1 ms early baseline end: replaced with a half-open baseline (dateRangeEndInclusive: false), so no sub-millisecond rows are dropped and the reported end is the queried end.
  • 🔵 Web app Event patterns: it runs drain3 in Pyodide with its own config, so rather than widen this PR I narrowed the changeset to the MCP and CLI paths.
  • P2 product-wide default: intended, so MCP event_patterns, MCP denoise and the CLI mine and match with the same masks; the default is covered by drain-masking.test.ts ("is on by default" plus the minePatterns cases). Happy to make it opt-in per caller if you'd prefer.

Tests: 489 api MCP unit tests, emergingSignalsTool.int.test.ts 5/5 and the pattern/denoise cases in queryTool.int.test.ts 13/13 pass; tsc, eslint and prettier clean.

…, UUID and hex masks

\b doesn't match between _ and a digit, so backup_2026-11-04 and span_<hex>
kept their dates and ids. Use alphanumeric lookarounds instead.
@bcanvural

Copy link
Copy Markdown
Contributor Author

Round 2:

  • Fixed: the bare date, time, UUID and hex masks used \b, which doesn't match between _ and a digit, so backup_2026-11-04 and span_<hex> kept their dates and ids. They now use alphanumeric lookarounds; tests added for both (and a req_<uuid> case).
  • Not changed, to keep the scope: splitting WindowOptions into fetch/mine types, and reading templates through a Drain.getCluster instead of match(). Happy to do either if you prefer.
  • Body:"…" snippets compiling to exact = predates this PR and is what fix(mcp): make event_patterns drill-down snippets executable #3426 reworks.

That's two automated rounds, so per AGENTS.md I'll stop here. @brandon-pereira, ready for a look when you have time.

…n as new_variant

With one Drain tree, a current-window message similar to a baseline one (same
length, enough shared tokens) is merged into the baseline cluster and only
generalizes its template, so it was no longer reported as emerging. Snapshot
each baseline template before mining the current window; where the current
window changed it, the current rows that fit the new template but not the old
one are reported as status new_variant, with the old template and a sample.
Read each cluster's final template by id (Drain.getCluster) instead of
re-matching the sample.
@bcanvural

Copy link
Copy Markdown
Contributor Author

One more round, out of the usual two, because the latest review found a regression this PR introduced: with one Drain tree, a new message that resembles a baseline one (e.g. user login failed: account locked next to user login failed: invalid password) was merged into the baseline pattern and stopped being reported. Fixed in 80ce437:

  • Each baseline cluster's template is recorded before the current window is mined. Where the current window changed it, the current rows that fit the new template but not the old one are reported in emerging with status: "new_variant", the old template as baselinePattern, a new line as sample, and their share as variantShare. Masked dates and ids don't change templates, so dated lines still don't show up.
  • Final templates are read by cluster id (Drain.getCluster) instead of re-matching the sample (the two minors on this).
  • Tests (emergingSignalsOrder.test.ts): the login example is reported as new_variant; the same dated line on two days gives nothing; identical windows give nothing; an account that was constant in the baseline and different now gives one new_variant entry instead of a "new" + "gone" pair (a real change). The order test now uses that case, so it would fail if mining followed query completion order. 492 MCP unit tests, the emerging_signals int tests (5/5) and the pattern/denoise int cases (13/13) pass.

Replied rather than changed: the Body:"…" whereSnippet form (pre-existing, reworked by #3426), and the web app's Pyodide drain3 config (out of scope; the changeset says the UI view is unchanged).

I'll stop here. @brandon-pereira, ready for a look when you have time.

…oves to its own PR

Review showed that comparing both windows in one Drain tree needs new-variant
detection of its own, so it moves to a follow-up. This PR keeps the default
masks (now also ctime/Java, log4j, Spark, klog and Go dates; drain3 JSON configs
without masks get none), the whereSnippet fix for tokens that contain a mask,
and their tests. emerging_signals goes back to two separate trees.
@bcanvural

Copy link
Copy Markdown
Contributor Author

I've narrowed this PR to masking (option 1 of #3544): the default masks, the whereSnippet fix for tokens that contain a mask, and their tests. The latest review showed that comparing both windows in one Drain tree needs new-variant detection of its own (a new message similar to a baseline one gets merged into it), which made this PR much bigger, so that part moved to #3556, with Draft PR #3557 stacked on this one. emerging_signals here keeps the two separate trees, as on main.

Diff against main is now 5 files, +225/−12. Tests: 3,065 common-utils, 486 MCP unit, and the emerging_signals + pattern/denoise integration tests pass.

@bcanvural bcanvural changed the title fix(patterns): mask timestamps and ids before Drain, mine emerging_signals windows in one tree fix(patterns): mask timestamps and ids before Drain Nov 4, 2026
@bcanvural

Copy link
Copy Markdown
Contributor Author

On the remaining 🟠 (the whereSnippet phrase still compiles to an exact Body:"…" match): that exact-match behaviour predates this PR, and it's what #3426 reworks. This PR only stops in-token masks like trace_id=<*> from being copied into the snippet. If you'd rather #3426 handle snippets entirely, I'm happy to drop that part from here.

The deep-review P2s (a fromJSON pass-through test, the backslash-escape test, 16+ digit decimals matching the hex mask) are small; I'll leave them for your call rather than start another round.

Take ClickHouse#3426's drill-down predicate in runEventPatterns.ts; drop this PR's
buildWhereSnippet and its test, which ClickHouse#3426 supersedes.
…n the drill-down regex

ClickHouse#3426's predicate matches a whole-token <*> as exactly one word and escapes
anything else literally, so a masked timestamp that spans spaces
(2026-11-04 06:12:12,345) or a mask inside a token ([<*>], trace_id=<*>)
matched nothing. Named masks keep Drain's clustering as it is (a mask is one
token that is equal across lines), and the regex maps <TS> to any text and
<ID> to one word, also inside a token. Whole-token <*> is unchanged.
@bcanvural

Copy link
Copy Markdown
Contributor Author

Merged main now that #3426 has landed (fast-forward, no force-push):

  • The conflict was the drill-down snippet. I took fix(mcp): make event_patterns drill-down snippets executable #3426's predicate and dropped this PR's buildWhereSnippet and its test.
  • fix(mcp): make event_patterns drill-down snippets executable #3426's templateToRegex treats a whole-token <*> as one word and escapes everything else, so masked patterns matched nothing: a masked timestamp can span spaces (2026-11-04 06:12:12,345), and a mask can sit inside a token ([<*>], trace_id=<*>). So the default masks are now named, <TS> for timestamps and <ID> for UUIDs/hex ids, and the regex maps <TS> to any text and <ID> to one word (also inside a token). A whole-token <*> is unchanged.
  • Named rather than <*> because Drain counts <*> positions against similarity: token-per-word <*> masks split lines with long dates (RFC-1123, ctime) into one pattern per value.

New tests mine dated Java, access-log, trace_id= and syslog lines with the default masks and check that each predicate matches its own samples and no other pattern's; they fail on main's regex. The same predicates matched exactly their own samples in ClickHouse.

Comment thread .changeset/patterns-mask-dated-lines.md Resolved
Mask the date parts of a timestamp to <DATE> and leave times and ids to
Drain, so a masked token compares like the date it replaced: equal within
a day, equal across days. Fixes the drill-down for a <*> merged over a
masked timestamp, and denoise now masks bodies like the miner does.
@bcanvural bcanvural changed the title fix(patterns): mask timestamps and ids before Drain fix(patterns): mask dates before Drain Nov 5, 2026
@bcanvural

Copy link
Copy Markdown
Contributor Author

The latest commit masks only dates: the date parts of a timestamp become <DATE>. Every line of a day shares its date and <DATE> matches itself the same way, so within one day lines group as without masks (tested), and across days the templates are the same. <TS>/<ID> are gone.

Why not times and ids: they already differ from line to line, so Drain turns them into <*> as soon as a pattern has two lines, and they don't cause the split. Dates are what does: every line of a day shares one, so Drain keeps it as a literal, and it changes the next day. Masking ids anyway costs more than it fixes:

  • As a named mask (<ID>), every id compares as equal, so Drain merges lines it keeps apart today, and a <*> it puts over <ID> broke the drill-down (the 🟠 below).
  • As <*>, Drain counts the token as a difference, so lines that are mostly ids or timestamps (e.g. <trace id> ok) never merge. Each forms its own pattern with the same text, e.g. <*> <*> ok. emerging_signals then adds those patterns together, and denoise can't tell them apart.

What's left: a line seen only once keeps its id or time in its template, so it can still show as new and gone.

  • 🟠 Drill-down: a mask now stays inside one word, so a <*> that Drain puts over <DATE> stands for one word, as templateToRegex expects. <DATE> matches one or more non-whitespace characters, also inside a token (backup_<DATE>.tar). Test: a dated line next to Started job at boot; each predicate matches all its samples.
  • 🟠 Denoise: buildNoisyBodyMatcher masks each body with the masker minePatterns uses (createPatternMasker) before comparing. Test with dated noisy lines.
  • 🔵 Changeset: adds @hyperdx/cli, and says the web app's Event patterns (drain3 in Pyodide, its own config) is unchanged.
  • P2 (16+ digit decimals matching the hex mask): the id masks are removed.
  • Testing gap: added the fromJSON round-trip test.
  • P3s: the semantics string uses the <DATE> constant, the regex now splits on the plain token string, and the word boundaries are named constants. Not done: a note on log lines that literally contain <DATE>; they fall under the documented superset.

One change within a day: klog lines with the same message but different severity letters are no longer merged, because I<DATE> and E<DATE> route by their letter.

On the review of this commit:

@brandon-pereira, ready for a look when you have time.

@bcanvural
bcanvural marked this pull request as draft November 6, 2026 12:34
@bcanvural

Copy link
Copy Markdown
Contributor Author

Moving this to draft to stay within the open-PR limit and make room for the fix for #3580 (PromQL dashboard variables fail on ClickHouse 26.10), which is more urgent for us. I'll mark this ready again when a slot frees.

Main moved buildNoisyBodyMatcher into common-utils and shares it with the
app, whose drain3 miner has no masks. Mask the body there only when asked
(masked: true), which the API's denoise passes.
@bcanvural
bcanvural marked this pull request as ready for review November 11, 2026 17:07
@bcanvural
bcanvural requested a review from a team as a code owner November 11, 2026 17:07
@bcanvural

Copy link
Copy Markdown
Contributor Author

Merged main again: #3621 moved buildNoisyBodyMatcher into common-utils and shares it with the app, whose drain3 miner has no masks. Masking the body is now opt-in (masked: true), and only the MCP denoise passes it. A test covers each side.

On the latest review, not changing:

  • fromJSON and perWord: nothing builds a config from JSON today, and the default masks don't go through it.
  • The rare line in the denoise test: right, the token count already tells it apart. The test is there for the dated noisy lines, which fail without masking.
  • Web app unaffected: it mines with drain3 in Pyodide, not with this config, and the new noisy-matcher test covers its unmasked path.
  • </> in the drill-down, and a literal <DATE> in a log line: both follow drain3's defaults, and a literal <DATE> only widens the drill-down filter, as before.

@bcanvural

Copy link
Copy Markdown
Contributor Author

@alex-fedotyev thanks for flagging this in #3662. #3551 merges cleanly with #3662 and conflicts with #3663 in patternDrilldown.ts, as expected. If your stack lands first, I'll move the <DATE> handling into templateToRegex in pattern-predicate.ts behind an option, since the app's drain3 miner doesn't mask dates and its templates shouldn't treat a literal <DATE> as a wildcard. If you'd rather land #3551 first, it's ready for review.

…ed-lines

# Conflicts:
#	packages/api/src/mcp/tools/query/patternDrilldown.ts
t === PATTERN_WILDCARD ? `[^${separators}]+` : escapeRe2Literal(t),
t === PATTERN_WILDCARD
? nonSeparators
: t.split(DATE_TOKEN).map(escapeRe2Literal).join(nonSeparators),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — templateToRegex now treats <DATE> as a wildcard for every caller, including templates mined without masks

Make the wildcard opt-in through a PatternRegexOptions field (e.g. masks: [DATE_MASK]) that only the MCP drill-down passes. The app's drain3 miner applies no masks, so in its templates a literal <DATE> from a log body (such as expected format <DATE>) is plain text. buildPatternPredicate now widens that literal to [^seps]+. This would also limit the compact/parse mismatch above to callers that ask for it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d590e7d: <DATE> is now behind the masks option, and only the MCP drill-down passes it.

@pulpdrew

Copy link
Copy Markdown
Contributor

Hey @bcanvural, apologies this took so long to get to.

#3663 has landed, so I merged main into this branch. The handling now lives in templateToRegex in pattern-predicate.ts, but unconditionally, so it still needs the opt-in you described. Before merging, please check the following:

  • Gate the wildcard behind a PatternRegexOptions field (e.g. masks: [DATE_MASK]) that only the MCP drill-down passes.
  • Honor the same option in compactRegex, or reject it there. Otherwise buildCompactPatternPredicate and buildPatternPredicate disagree for any caller that sets it.
  • Move the templateToRegex tests from api's patternDrilldown.test.ts to pattern-predicate.test.ts, now that the function lives in common-utils.
  • Export DATE_TOKEN from common-utils rather than defining it in both pattern-predicate.ts and patternDrilldown.ts.

Feel free to ping me when ready for re-review!

…ller passes them

The MCP drill-down passes masks: [DATE_MASK]; the app's predicates keep
<DATE> literal. compactRegex honors the option too, so both builders agree.
DATE_TOKEN is exported from common-utils, and the <DATE> regex tests move to
pattern-predicate.test.ts.
for (const mi of instructions) {
allowedPatterns.push(mi.pattern);
// A per-word mask stands for one word of its pattern's match.
allowedPatterns.push(mi.perWord ? '\\S+' : mi.pattern);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — With exactMatching, a per-word \S+ alternative lets <DATE> match any word

The <DATE> capture group joins every DATE pattern into one alternation, and the per-word ones add \S+, which swallows the rest. So extractParameters('<DATE> x', 'hello x', true) now returns 'hello' instead of null. Use one per-word sub-pattern per instruction (e.g. the instruction's single-word forms) or skip validation for per-word masks explicitly, and add an exactMatching=true test.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this one: extractParameters with exactMatching has no callers in the repo (it came with the drain3 port). A per-word mask stands for one word of a longer match, such as the Nov of Nov 4, so no single-word pattern can check it exactly; \S+ is the closest.

@bcanvural

Copy link
Copy Markdown
Contributor Author

Thanks for merging main, @pulpdrew! Addressed in d590e7d:

  • <DATE> handling is now behind a masks option on PatternRegexOptions (masks: [DATE_MASK]), and only the MCP drill-down passes it. Without it, <DATE> is literal, so the app's predicates are as on main.
  • compactRegex uses the same token helper, so both builders agree; a test checks that for dated templates.
  • The <DATE> tests moved to pattern-predicate.test.ts, with one more for the literal case.
  • DATE_TOKEN is exported from common-utils and patternDrilldown.ts imports it.

One point from the deep review: TemplateMinerConfig now applies the default masks, and the changeset marks @hyperdx/common-utils as a patch. Every caller in the repo wants the masks (the app's drain3 miner doesn't use this class), but if you'd rather ship it as a minor bump, I'll change the changeset.

Ready for another look.

pulpdrew
pulpdrew previously approved these changes Nov 19, 2026

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the contribution

@kodiakhq
kodiakhq Bot merged commit d99be76 into ClickHouse:main Nov 19, 2026
26 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants