Skip to content

sqlmesh test should support --use-project-index for scoped loads #6024

Open

Description

@cmgoffena13

Summary

sqlmesh lint --use-project-index --model xxx loads only the selected models and their upstream dependencies. sqlmesh test always fully loads the project, then filters which tests run. For a pre-commit hook (or sqlmesh test models/orders.sql), parsing every model is the dominant cost. Test YAML discovery is cheap; model load is not.

The project index is a loader feature. Lint grew the flag first. Unit tests need the same subgraph: the model under test plus transitive upstream for schema resolution. They do not need downstream models.

Current behavior

  • The CLI group callback constructs Context with load=True before test runs
  • Context.test() only filters self._model_test_metadata
  • linter.use_project_index does not apply to test (and should not; that config key is the wrong place)

Proposed CLI

sqlmesh test --local --use-project-index models/orders.sql
sqlmesh test --local --use-project-index tests/test_items.yaml
sqlmesh test --local --use-project-index models/a.sql tests/test_b.yaml

Load order should match lint:

  1. Parse args (model paths, test YAML paths, --select-model)
  2. Resolve to model FQNs (index maps relative path → FQN; test YAML has model:)
  3. load(model_fqns=…, use_project_index=True)
  4. Run the union of those tests

No paths / no --select-model still loads everything and refreshes the index (same as sqlmesh lint with no --model ).

Missing or stale index falls back to a full load and rebuilds it. Cross-project dependencies already force that fallback.

Config

Do not reuse linter.use_project_index. CLI flag first. A top-level use_project_index later is fine. Probably best to add in a flag specific for test as well for granular control.

Acceptance

  • sqlmesh test --use-project-index models/a.sql does not parse unrelated model files when the index is warm
  • Upstream of a is loaded so SELECT * / column resolution still works
  • A selected test whose model is missing after a scoped load errors, does not warn-and-skip (create_test currently returns None and the suite can pass)
  • Bare sqlmesh test --use-project-index still runs the full suite
  • --local and --use-project-index compose; one does not imply the other

Extra

The baseline above still calls load_model_tests(), which globs every tests/test*.yaml, YAML-parses the full body (including fixture rows, with variable substitution), and keeps it all in memory. For a big suite, a staged models/a.sql still opens every test file even when only a's tests will run.

Would be really nice for big projects: put the join in the project index, not the fixtures.

  • Store model FQN → test file paths (staged model → open only those YAMLs)
  • Store test file path → model FQNs (staged YAML → load those models + upstream)
  • Do not cache input/output rows; those load when the test actually runs
  • Invalidate when tests/ changes (file set + mtimes), including a model: retarget, and bump the index version

Then sqlmesh test --local --use-project-index models/a.sql skips state, unrelated SQL, and unrelated test YAML. Wiring the flag without this extra still reads the whole suite.

Not required for the first cut. Optional follow-on in this issue if the index work is already in flight.

Activity

  1. cookieark7 commented on Nov 20, 2026

    @cookieark7
    Contributor

    I'd like to pick this one up.

    I tried it on current main with a DuckDB project of about 2,000 models, where a has one upstream model and one unit test:

    Models loaded Time
    Full load (warm cache) 2,002 1.49s
    Scoped load via the index 2 0.26s
    sqlmesh lint --local --use-project-index --model demo.a 0.98s
    sqlmesh test --local tests/test_a.yaml 2.39s

    It also isn't only about speed. With one unrelated model broken, the scoped lint still passes, but sqlmesh test fails while loading that unrelated file. And a test whose model doesn't exist only warns and exits 0, which matches what you described for create_test.

    What I'm planning for the first cut, without the "Extra" part:

    • Add --use-project-index to test, and skip the CLI's initial load the same way lint does when the flag is passed.
    • Resolve model file paths through the index, and test YAML paths through their model: key, then load those models plus upstream.
    • Error, rather than warn and skip, when a selected test's model isn't loaded in a scoped run.
    • Keep a bare sqlmesh test --use-project-index as a full run, and keep it independent from --local.

    I'll start on it today.

  2. added a commit that references this issue on Nov 21, 2026
    b5e310e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions