Skip to content

RFC: Export SQLMesh metadata as dbt-compatible artifacts (manifest.json) for ecosystem integration #5821

Open

Description

@nickmuoh

Base on @StuffbyYuki Slack message:

Just a random thought - I wonder how hard it'd be to make sqlmesh dbt-integrated tool compatible where sqlmesh can utilize the kind of integrations dbt has with other tools (BI, data observability, etc) via dbt artifacts. I can see some people are hesitant to try sqlmesh because they already have tools integrated around dbt, so if sqlmesh can use the existing dbt ecosystem, I can see more users would be willing to try sqlmesh.

Problem Statement

A major friction point for users considering adopting—or staying with—SQLMesh is the loss of the broader dbt ecosystem. Many organizations have built extensive data stacks around dbt using data observability platforms, BI tools, and data cataloging solutions that rely heavily on dbt artifacts (such as manifest.json).

If a team starts a project natively in SQLMesh, they currently cannot utilize this vast, pre-existing ecosystem of integrations. Waiting for every third-party data tool to build and maintain native SQLMesh integrations is a slow process and creates a significant bottleneck for adoption. In some cases, this lack of ecosystem support is causing users who have already adopted SQLMesh to consider migrating back to dbt just to regain access to their preferred observability and BI integrations.

Impact / Use Cases

Users are currently blocked from easily integrating SQLMesh with:

  • Data Observability: Tools like Monte Carlo, Elementary, or Metaplane that ingest metadata and lineage to monitor data health.
  • Data Catalogs: Platforms like Atlan, DataHub, or Select that rely on manifest files to map organizational data assets.
  • BI Tools: Dashboards that dynamically update or expose lineage based on transformation tool metadata.

Additional Context

We need a way to bridge this gap so SQLMesh users can utilize the existing dbt ecosystem without being blocked by third-party vendor roadmaps. What exact capabilities and integration points would we need to support to alleviate this ecosystem lock-in?

Activity

  1. mday-io commented on Jun 1, 2026

    @mday-io
    Collaborator

    Could be interesting. Might be something to handle integration by integration until we see patterns in required metadata?

  2. dcohen24 commented on Jun 3, 2026

    @dcohen24

    We're working on a sqlmesh plugin for datahub, with acryl at the moment, btw

  3. StuffbyYuki commented on Jun 4, 2026

    @StuffbyYuki
    Collaborator

    Which one do we think makes more sense? Build native integrations one by one like we already have with tools like dlt, dagster, and soon datahub or try building dbt compatible artifacts

  4. mday-io commented on Jun 4, 2026

    @mday-io
    Collaborator

    It's certainly worth attempting the dbt manifest adapter. Waiting for the community to build connector by connector will stop many adopters. It's just a matter of what is possible and how much it'll take to maintain. Maybe since the dbt converter was already built that will help.

  5. dcohen24 commented on Jun 4, 2026

    @dcohen24
  6. tobymao commented on Jun 4, 2026

    @tobymao
    Contributor

    i recommend building the manifest that’s a huge unlock, you could always expose the physical table directly

  7. StuffbyYuki commented on Jun 26, 2026

    @StuffbyYuki
    Collaborator

    Does anybody want to take a stab at this? It doesn't have to be perfect and we can build incrementally

  8. c3us-dev commented on Jun 29, 2026

    @c3us-dev

    I’d be happy to help with this, though I want to be upfront that I’m still fairly fresh to this project and am still learning the ins and outs of SQLMesh internals so I would need to lean on feedback/guidance from more experienced contributors/maintainers.

    I recently ran into the same general hiccups/nuances @dcohen24 mentioned while experimenting with a possible SQLMesh to OpenMetadata bridge. My first thought was that it would be simplest to build something standalone/independent of either project: add row counts, replace the view definitions pulled by metadata ingestion with rendered SQLMesh logical model SQL, and translate SQLMesh lineage into OpenLineage/OpenMetadata-compatible API calls in a second pass.

    Before going too far down that path, I saw this issue/discussion and thought it would be worth experimenting with how far I could get letting OpenMetadata's native dbt ingestion do its thing and take a stab at emitting dbt-like artifacts that satisfied the bare minimum of what the OpenMetadata dbt integration needed with a small scratch example.

    summary of setup

    This metadata bridge was tested against SQLMesh 0.235.4 using the sushi example project and exported the 20ish SQLMesh models into a neutral, SQLMesh-detail preserving metadata shape, that in turn was used for emitting a dbt-like manifest.json, catalog.json, and run_results.json.

    what worked

    • OpenMetadata 1.12.6 accepted the generated dbt artifacts.
    • dbt ingestion replayed successfully: 44 dbt records, 20 OpenMetadata records, 0 warnings, 0 errors.
    • OpenMetadata attached DBT data-model metadata, compiled SQL, model paths/dbt Source Project, upstreams, and table-level DbtLineage.
    • Impact analysis worked (table-level only)
    • SQLMesh-specific state was preserved under meta.sqlmesh in the manifest.json, including logical relation, active physical relation, environment, snapshot id/version, kind, and fingerprint, for potential future use by adapted OpenMetadata profilers/metadata flows.

    gaps

    • One wrinkle that came up is that OpenMetadata dbt ingestion required matching database/table entities that already existed. Because of this, the artifacts really just provided an attachment layer for existing assets in this test more than a full ingestion path. The gap between the dbt metadata and the profiling/observability pipeline behavior that normally would of created the database service/table assets came up in a few different ways, but for this creation step I just used a small API bridge to create the table entities first.
    • The dbt bridge produced table-level lineage only, which really is just a limitation of dbt/current manifest contents it seems, but means that all the interesting/useful metadata I would need out of the "meta" catchall in the manifest.json artifact still requires adaption of the pipeline code on the OpenMetadata-side or some secondary scripts/pipelines still. I need to look into @nicosuave's sqlmesh-openlineage repo more closely, but for model-dialect aware lineage ingestion OpenMetadata uses SQLGlot as well with a couple fallbacks so it seems this can be readily tackled from either end.
    • SQLMesh’s logical/virtual alias layer still proved thorny and I wasn't sure if there was a better dbt-side approach to resolve this, but I ended up just dumping an "active_physical_relation" identifier string into the meta.sqlmesh catchall in manifest.json as well. The profilers failed if the logical alias was not queryable by the target service, even though actual execution might use the physical snapshot tables. To get around this I made some temporary, throwaway Postgres mirrors for queryable logical aliases, but that work around isn't much of a production solution across engines..

    takeaway

    Overall it seems the dbt-artifact bridge is definitely useful for catalogs, rendered SQL visibility, model metadata, and table-level linage, but in the context of a tool like OpenMetadata the logical vs. physical hurdle, more granular lineage, and profiling/observability functionality pretty quickly pushes the solution toward making use of the SQLMesh specific metadata that can be packaged in manifest.json but that still needs SQLMesh-aware connector/hook adaptions to be leveraged.

    It seems that flattening those aspects too aggressively for dbt-focused compatibility potentially leads to future integrations being more difficult/less expressive, but I'd love to hear from you all that are more familiar with the project design/internals if that neutral sqlmesh metadata output seems worthwhile or if that is an unnecessary additional layer/intermediate output and tool-specific metadata emitters should be wired up more directly for flexibility around what's included and structure. Similarly, does separating out column lineage into an OpenLineage/native metadata path look like the best way forward or is there another dbt-compatible artifact path still worth exploring further? Any other thoughts on the direction/structure of this would be appreciated as well as any feedback related to SQLMesh and dbt details I may have overlooked or mishandled. Thanks.

  9. krutoileshii commented on Jul 1, 2026

    @krutoileshii

    Personally, I'd love to see something similar, but for me it's around docglow. Docglow already uses sqlglot so figure it would be easy to just generate required manifest for the site + it would give this projects docs similar to dbt with some modifications.

    https://www.docglow.com/

  10. StuffbyYuki commented on Jul 1, 2026

    @StuffbyYuki
    Collaborator

    So for this specific sqlmesh feature, we shouldn't go too deep trying to make things "sqlmesh grade" and keep it simple. And areas we see a benefit adding sqlmesh metadata and whatnot, we'll add our own extension or extend the dbt integration?

  11. c3us-dev commented on Jul 2, 2026

    @c3us-dev

    @StuffbyYuki fair enough - do you have a sense of what some of the most popular/valuable dbt integrations are that I can/should use to validate a basic first pass at emitting the main dbt artifacts?

    @krutoileshii docglow looks very interesting; I have seen people mention the dbt project doc generation as one of the main factors that led to them ultimately pivoting back to dbt after seriously considering sqlmesh so it could be a good add in this same vein of reducing friction around trial and adoption

  12. krutoileshii commented on Jul 2, 2026

    @krutoileshii

    @c3us-dev for us the big reason for sticking with dbt for prod are currently below:

    Docs - we have a lot of end users and having a simple doc site is a must. can it be done outside of codebase, sure. but we all know how much everyone loves documenting.
    Observability - Specifically around audits. having issues logged and tracking them over time is huge. I think for sqlmesh this would be easier to tackle as there is already a metadata database involved.
    Catalog - Open metadata/Datahub integration of some sorts.
    Orchestrator Integration: We heavily use airflow and having the scheduling and detailed dags in there would be great. (minor).

    Other than that, it really is easier to get going and understand vs dbt.

    Really hoping our tests go well overall as if ultimately like to migrate the team over at some point.

    If I could have it my way, Dox site would be my #1 ask.

  13. StuffbyYuki commented on Jul 2, 2026

    @StuffbyYuki
    Collaborator

    @c3us-dev I think we need to do some analysis on which tools use only the manifest.json or manifest.json + other dbt artifacts. And maybe we categorize them and decide to what to tackle first. We might be able to go pretty far just building the manifest.json.

  14. StuffbyYuki commented on Jul 2, 2026

    @StuffbyYuki
    Collaborator

    @krutoileshii I feel you! The docs feature is something we'll def do and will be a huge add. Since we have richer metadata, we can probably make it better than what dbt offers too.

  15. krutoileshii commented on Jul 2, 2026

    @krutoileshii

    I find docglow hi is pretty good for lineage and presentation, you could probably strip docglow down since sqlmesh has its own API that's much easier that. parsing manifests and stuff

  16. changed the title [-]Export SQLMesh metadata as dbt-compatible artifacts `(manifest.json)` for ecosystem integration[/-] [+]RFC: Export SQLMesh metadata as dbt-compatible artifacts `(manifest.json)` for ecosystem integration[/+] on Sep 23, 2026
  17. bosprimigenious commented on Nov 3, 2026

    @bosprimigenious

    Following the question about which consumers need more than manifest.json, I checked two consumer loaders at pinned revisions:

    Consumer / scope Artifact dependency visible in the source
    OpenMetadata, HTTP dbt ingestion The manifest is fetched and parsed; catalog, run-results and sources are conditional. Its object parser also accepts those three as absent.
    dbt Docs, site bootstrap The loader requests both manifest.json and catalog.json; it does not request run_results.json.

    Sources: OpenMetadata HTTP loader, optional object parsing, and dbt Docs bootstrap.

    One useful distinction for the first acceptance fixture: OpenMetadata only enters its dbt-test processing branch when run-results are present (source). A manifest-only metadata fixture and an audit-result fixture therefore test different capabilities; the latter should use actual execution results rather than placeholder successes.

    My suggestion is to validate manifest-only metadata ingestion first, then add a manifest+catalog docs fixture, keeping audit-results support separate. This is a source-level contract check, not a claim that a SQLMesh-generated manifest already passes either consumer's schema or integration. I have not run those integrations.

    AI assistance: researched and checked with OpenAI Codex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    FeatureAdds new functionality

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions