Skip to content

Model loading hangs: DIALECT_PATTERN backtracks exponentially over block comments when a MODEL block has no dialect #6203

Open

Description

@PTruscott

Description

Loading a project hangs, with no error, when a model's MODEL block names no dialect and the file contains a run of /* */ comments after a comma or bracket.

parse() searches the first MAX_MODEL_DEFINITION_SIZE characters of every file with DIALECT_PATTERN before parsing it. It does this even when default_dialect is set, so a project-wide model_defaults.dialect doesn't avoid it. Since #5651 the pattern allows comments between tokens with:

WS_OR_COMMENT = r"(?:\s|--[^\n]*\n|/\*.*?\*/)"

Under re.DOTALL, the lazy /\*.*?\*/ can still stretch past its own */ to a later one, so a run of n block comments can be matched in about 2ⁿ⁻¹ ways. When the file has no dialect key the search fails, so it tries all of them, at every ( or , it reaches.

Reproduction

The pattern first shipped in 0.228.2 and is unchanged in 0.236.4 and on main. Run with 0.236.0:

import time

from sqlmesh.core.dialect import DIALECT_PATTERN

for n in range(16, 25, 2):
    comments = "".join(f"  /* note {i} */\n" for i in range(n))
    sql = f"MODEL (\n  name db.example,\n  kind FULL\n);\n\nSELECT\n  a,\n{comments}  b\nFROM t\n"
    start = time.perf_counter()
    DIALECT_PATTERN.search(sql)
    print(f"{n} comments: {time.perf_counter() - start:.2f}s")
16 comments: 0.02s
18 comments: 0.07s
20 comments: 0.30s
22 comments: 1.28s
24 comments: 4.92s

Each extra comment roughly doubles the time. In our project, a real model never finished loading, so every command that loads the project hung.

Why naming dialect in each model isn't enough

Adding dialect to the MODEL block avoids the hang, because the search then matches near the top of the block. But it can't be relied on:

  • Leaving it out is the documented default. Every project must set model_defaults.dialect, and the docs say a model's dialect falls back to it. Most models never name one, so most models take the slow path.
  • What triggers it isn't visible in review. Whether a file hangs depends on how many block comments follow a comma or bracket in its first 10,000 characters. Documenting a model with a few more comments is enough to stop the project loading.
  • It fails silently, for the whole project. There is no error or timeout. Every command that loads the project (plan, run, tests, CI, scheduled jobs) hangs, not just the one model.
  • It has to be done in advance, everywhere. To be safe, every model and audit would need a dialect line before its first run of comments, and every new file would need it too. That repeats the project default in every file to dodge a parsing bug.

Suggested fix

Match block comments balanced, so each */ closes exactly one /* and a run of comments has only one way to match. SQLGlot reads block comments as nested, so the pattern matches nesting a few levels deep rather than not at all:

BLOCK_COMMENT = r"/\*(?:[^*]|\*(?!/))*\*/"
for _ in range(2):
    BLOCK_COMMENT = rf"/\*(?:[^*/]|\*(?!/)|/(?!\*)|{BLOCK_COMMENT})*\*/"
WS_OR_COMMENT = rf"(?:\s|--[^\n]*\n|{BLOCK_COMMENT})"

With this change, the 40-comment case and our real model finish in under a millisecond, and worst cases filling the whole 10,000-character window with commas and comments take about a millisecond. Across 363 SQL files (our project's models and audits, plus every .sql file under this repo's examples/ and tests/), it returns the same dialect as the current pattern for every file the current pattern finishes on. It also does for nested comments up to three levels deep. A deeper nest falls back to the default dialect: a regex can't match arbitrary nesting, and the current pattern only handles it by stretching across comments, which is what causes the hang.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions