You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
RFC: Export SQLMesh metadata as dbt-compatible artifacts (manifest.json) for ecosystem integration #5821
Just a random thought - I wonder how hard it'd be to make sqlmesh dbt-integrated tool compatible where sqlmesh can utilize the kind of integrations dbt has with other tools (BI, data observability, etc) via dbt artifacts. I can see some people are hesitant to try sqlmesh because they already have tools integrated around dbt, so if sqlmesh can use the existing dbt ecosystem, I can see more users would be willing to try sqlmesh.
Problem Statement
A major friction point for users considering adopting—or staying with—SQLMesh is the loss of the broader dbt ecosystem. Many organizations have built extensive data stacks around dbt using data observability platforms, BI tools, and data cataloging solutions that rely heavily on dbt artifacts (such as manifest.json).
If a team starts a project natively in SQLMesh, they currently cannot utilize this vast, pre-existing ecosystem of integrations. Waiting for every third-party data tool to build and maintain native SQLMesh integrations is a slow process and creates a significant bottleneck for adoption. In some cases, this lack of ecosystem support is causing users who have already adopted SQLMesh to consider migrating back to dbt just to regain access to their preferred observability and BI integrations.
Impact / Use Cases
Users are currently blocked from easily integrating SQLMesh with:
Data Observability: Tools like Monte Carlo, Elementary, or Metaplane that ingest metadata and lineage to monitor data health.
Data Catalogs: Platforms like Atlan, DataHub, or Select that rely on manifest files to map organizational data assets.
BI Tools: Dashboards that dynamically update or expose lineage based on transformation tool metadata.
Additional Context
We need a way to bridge this gap so SQLMesh users can utilize the existing dbt ecosystem without being blocked by third-party vendor roadmaps. What exact capabilities and integration points would we need to support to alleviate this ecosystem lock-in?
Which one do we think makes more sense? Build native integrations one by one like we already have with tools like dlt, dagster, and soon datahub or try building dbt compatible artifacts
It's certainly worth attempting the dbt manifest adapter. Waiting for the community to build connector by connector will stop many adopters. It's just a matter of what is possible and how much it'll take to maintain. Maybe since the dbt converter was already built that will help.
Definitely worth it
But there will be nuances... The view level of abstraction will mean a data
observability platform relying on it, won't find row counts or freshness in
information schema, for instance
The " native lineage" that might show up will be pointing from the views to
the backing tables, etc.
I think you're right, its worth picking up... But the use cases may not
work as perfectly as we might expect
I’d be happy to help with this, though I want to be upfront that I’m still fairly fresh to this project and am still learning the ins and outs of SQLMesh internals so I would need to lean on feedback/guidance from more experienced contributors/maintainers.
I recently ran into the same general hiccups/nuances @dcohen24 mentioned while experimenting with a possible SQLMesh to OpenMetadata bridge. My first thought was that it would be simplest to build something standalone/independent of either project: add row counts, replace the view definitions pulled by metadata ingestion with rendered SQLMesh logical model SQL, and translate SQLMesh lineage into OpenLineage/OpenMetadata-compatible API calls in a second pass.
Before going too far down that path, I saw this issue/discussion and thought it would be worth experimenting with how far I could get letting OpenMetadata's native dbt ingestion do its thing and take a stab at emitting dbt-like artifacts that satisfied the bare minimum of what the OpenMetadata dbt integration needed with a small scratch example.
summary of setup
This metadata bridge was tested against SQLMesh 0.235.4 using the sushi example project and exported the 20ish SQLMesh models into a neutral, SQLMesh-detail preserving metadata shape, that in turn was used for emitting a dbt-like manifest.json, catalog.json, and run_results.json.
what worked
OpenMetadata 1.12.6 accepted the generated dbt artifacts.
OpenMetadata attached DBT data-model metadata, compiled SQL, model paths/dbt Source Project, upstreams, and table-level DbtLineage.
Impact analysis worked (table-level only)
SQLMesh-specific state was preserved under meta.sqlmesh in the manifest.json, including logical relation, active physical relation, environment, snapshot id/version, kind, and fingerprint, for potential future use by adapted OpenMetadata profilers/metadata flows.
gaps
One wrinkle that came up is that OpenMetadata dbt ingestion required matching database/table entities that already existed. Because of this, the artifacts really just provided an attachment layer for existing assets in this test more than a full ingestion path. The gap between the dbt metadata and the profiling/observability pipeline behavior that normally would of created the database service/table assets came up in a few different ways, but for this creation step I just used a small API bridge to create the table entities first.
The dbt bridge produced table-level lineage only, which really is just a limitation of dbt/current manifest contents it seems, but means that all the interesting/useful metadata I would need out of the "meta" catchall in the manifest.json artifact still requires adaption of the pipeline code on the OpenMetadata-side or some secondary scripts/pipelines still. I need to look into @nicosuave's sqlmesh-openlineage repo more closely, but for model-dialect aware lineage ingestion OpenMetadata uses SQLGlot as well with a couple fallbacks so it seems this can be readily tackled from either end.
SQLMesh’s logical/virtual alias layer still proved thorny and I wasn't sure if there was a better dbt-side approach to resolve this, but I ended up just dumping an "active_physical_relation" identifier string into the meta.sqlmesh catchall in manifest.json as well. The profilers failed if the logical alias was not queryable by the target service, even though actual execution might use the physical snapshot tables. To get around this I made some temporary, throwaway Postgres mirrors for queryable logical aliases, but that work around isn't much of a production solution across engines..
takeaway
Overall it seems the dbt-artifact bridge is definitely useful for catalogs, rendered SQL visibility, model metadata, and table-level linage, but in the context of a tool like OpenMetadata the logical vs. physical hurdle, more granular lineage, and profiling/observability functionality pretty quickly pushes the solution toward making use of the SQLMesh specific metadata that can be packaged in manifest.json but that still needs SQLMesh-aware connector/hook adaptions to be leveraged.
It seems that flattening those aspects too aggressively for dbt-focused compatibility potentially leads to future integrations being more difficult/less expressive, but I'd love to hear from you all that are more familiar with the project design/internals if that neutral sqlmesh metadata output seems worthwhile or if that is an unnecessary additional layer/intermediate output and tool-specific metadata emitters should be wired up more directly for flexibility around what's included and structure. Similarly, does separating out column lineage into an OpenLineage/native metadata path look like the best way forward or is there another dbt-compatible artifact path still worth exploring further? Any other thoughts on the direction/structure of this would be appreciated as well as any feedback related to SQLMesh and dbt details I may have overlooked or mishandled. Thanks.
Personally, I'd love to see something similar, but for me it's around docglow. Docglow already uses sqlglot so figure it would be easy to just generate required manifest for the site + it would give this projects docs similar to dbt with some modifications.
So for this specific sqlmesh feature, we shouldn't go too deep trying to make things "sqlmesh grade" and keep it simple. And areas we see a benefit adding sqlmesh metadata and whatnot, we'll add our own extension or extend the dbt integration?
@StuffbyYuki fair enough - do you have a sense of what some of the most popular/valuable dbt integrations are that I can/should use to validate a basic first pass at emitting the main dbt artifacts?
@krutoileshii docglow looks very interesting; I have seen people mention the dbt project doc generation as one of the main factors that led to them ultimately pivoting back to dbt after seriously considering sqlmesh so it could be a good add in this same vein of reducing friction around trial and adoption
@c3us-dev for us the big reason for sticking with dbt for prod are currently below:
Docs - we have a lot of end users and having a simple doc site is a must. can it be done outside of codebase, sure. but we all know how much everyone loves documenting.
Observability - Specifically around audits. having issues logged and tracking them over time is huge. I think for sqlmesh this would be easier to tackle as there is already a metadata database involved.
Catalog - Open metadata/Datahub integration of some sorts.
Orchestrator Integration: We heavily use airflow and having the scheduling and detailed dags in there would be great. (minor).
Other than that, it really is easier to get going and understand vs dbt.
Really hoping our tests go well overall as if ultimately like to migrate the team over at some point.
If I could have it my way, Dox site would be my #1 ask.
@c3us-dev I think we need to do some analysis on which tools use only the manifest.json or manifest.json + other dbt artifacts. And maybe we categorize them and decide to what to tackle first. We might be able to go pretty far just building the manifest.json.
@krutoileshii I feel you! The docs feature is something we'll def do and will be a huge add. Since we have richer metadata, we can probably make it better than what dbt offers too.
I find docglow hi is pretty good for lineage and presentation, you could probably strip docglow down since sqlmesh has its own API that's much easier that. parsing manifests and stuff
changed the title [-]Export SQLMesh metadata as dbt-compatible artifacts `(manifest.json)` for ecosystem integration[/-][+]RFC: Export SQLMesh metadata as dbt-compatible artifacts `(manifest.json)` for ecosystem integration[/+]on Sep 23, 2026
One useful distinction for the first acceptance fixture: OpenMetadata only enters its dbt-test processing branch when run-results are present (source). A manifest-only metadata fixture and an audit-result fixture therefore test different capabilities; the latter should use actual execution results rather than placeholder successes.
My suggestion is to validate manifest-only metadata ingestion first, then add a manifest+catalog docs fixture, keeping audit-results support separate. This is a source-level contract check, not a claim that a SQLMesh-generated manifest already passes either consumer's schema or integration. I have not run those integrations.
AI assistance: researched and checked with OpenAI Codex.
Activity
mday-io commentedon Jun 1, 2026
Could be interesting. Might be something to handle integration by integration until we see patterns in required metadata?
dcohen24 commentedon Jun 3, 2026
We're working on a sqlmesh plugin for datahub, with acryl at the moment, btw
StuffbyYuki commentedon Jun 4, 2026
Which one do we think makes more sense? Build native integrations one by one like we already have with tools like dlt, dagster, and soon datahub or try building dbt compatible artifacts
mday-io commentedon Jun 4, 2026
It's certainly worth attempting the dbt manifest adapter. Waiting for the community to build connector by connector will stop many adopters. It's just a matter of what is possible and how much it'll take to maintain. Maybe since the dbt converter was already built that will help.
dcohen24 commentedon Jun 4, 2026
tobymao commentedon Jun 4, 2026
i recommend building the manifest that’s a huge unlock, you could always expose the physical table directly
StuffbyYuki commentedon Jun 26, 2026
Does anybody want to take a stab at this? It doesn't have to be perfect and we can build incrementally
c3us-dev commentedon Jun 29, 2026
I’d be happy to help with this, though I want to be upfront that I’m still fairly fresh to this project and am still learning the ins and outs of SQLMesh internals so I would need to lean on feedback/guidance from more experienced contributors/maintainers.
I recently ran into the same general hiccups/nuances @dcohen24 mentioned while experimenting with a possible SQLMesh to OpenMetadata bridge. My first thought was that it would be simplest to build something standalone/independent of either project: add row counts, replace the view definitions pulled by metadata ingestion with rendered SQLMesh logical model SQL, and translate SQLMesh lineage into OpenLineage/OpenMetadata-compatible API calls in a second pass.
Before going too far down that path, I saw this issue/discussion and thought it would be worth experimenting with how far I could get letting OpenMetadata's native dbt ingestion do its thing and take a stab at emitting dbt-like artifacts that satisfied the bare minimum of what the OpenMetadata dbt integration needed with a small scratch example.
summary of setup
This metadata bridge was tested against SQLMesh
0.235.4using the sushi example project and exported the 20ish SQLMesh models into a neutral, SQLMesh-detail preserving metadata shape, that in turn was used for emitting a dbt-likemanifest.json,catalog.json, andrun_results.json.what worked
1.12.6accepted the generated dbt artifacts.DbtLineage.meta.sqlmeshin themanifest.json, including logical relation, active physical relation, environment, snapshot id/version, kind, and fingerprint, for potential future use by adapted OpenMetadata profilers/metadata flows.gaps
manifest.jsonartifact still requires adaption of the pipeline code on the OpenMetadata-side or some secondary scripts/pipelines still. I need to look into @nicosuave's sqlmesh-openlineage repo more closely, but for model-dialect aware lineage ingestion OpenMetadata uses SQLGlot as well with a couple fallbacks so it seems this can be readily tackled from either end.meta.sqlmeshcatchall inmanifest.jsonas well. The profilers failed if the logical alias was not queryable by the target service, even though actual execution might use the physical snapshot tables. To get around this I made some temporary, throwaway Postgres mirrors for queryable logical aliases, but that work around isn't much of a production solution across engines..takeaway
Overall it seems the dbt-artifact bridge is definitely useful for catalogs, rendered SQL visibility, model metadata, and table-level linage, but in the context of a tool like OpenMetadata the logical vs. physical hurdle, more granular lineage, and profiling/observability functionality pretty quickly pushes the solution toward making use of the SQLMesh specific metadata that can be packaged in
manifest.jsonbut that still needs SQLMesh-aware connector/hook adaptions to be leveraged.It seems that flattening those aspects too aggressively for dbt-focused compatibility potentially leads to future integrations being more difficult/less expressive, but I'd love to hear from you all that are more familiar with the project design/internals if that neutral sqlmesh metadata output seems worthwhile or if that is an unnecessary additional layer/intermediate output and tool-specific metadata emitters should be wired up more directly for flexibility around what's included and structure. Similarly, does separating out column lineage into an OpenLineage/native metadata path look like the best way forward or is there another dbt-compatible artifact path still worth exploring further? Any other thoughts on the direction/structure of this would be appreciated as well as any feedback related to SQLMesh and dbt details I may have overlooked or mishandled. Thanks.
krutoileshii commentedon Jul 1, 2026
Personally, I'd love to see something similar, but for me it's around docglow. Docglow already uses sqlglot so figure it would be easy to just generate required manifest for the site + it would give this projects docs similar to dbt with some modifications.
https://www.docglow.com/
StuffbyYuki commentedon Jul 1, 2026
So for this specific sqlmesh feature, we shouldn't go too deep trying to make things "sqlmesh grade" and keep it simple. And areas we see a benefit adding sqlmesh metadata and whatnot, we'll add our own extension or extend the dbt integration?
c3us-dev commentedon Jul 2, 2026
@StuffbyYuki fair enough - do you have a sense of what some of the most popular/valuable dbt integrations are that I can/should use to validate a basic first pass at emitting the main dbt artifacts?
@krutoileshii docglow looks very interesting; I have seen people mention the dbt project doc generation as one of the main factors that led to them ultimately pivoting back to dbt after seriously considering sqlmesh so it could be a good add in this same vein of reducing friction around trial and adoption
krutoileshii commentedon Jul 2, 2026
@c3us-dev for us the big reason for sticking with dbt for prod are currently below:
Docs - we have a lot of end users and having a simple doc site is a must. can it be done outside of codebase, sure. but we all know how much everyone loves documenting.
Observability - Specifically around audits. having issues logged and tracking them over time is huge. I think for sqlmesh this would be easier to tackle as there is already a metadata database involved.
Catalog - Open metadata/Datahub integration of some sorts.
Orchestrator Integration: We heavily use airflow and having the scheduling and detailed dags in there would be great. (minor).
Other than that, it really is easier to get going and understand vs dbt.
Really hoping our tests go well overall as if ultimately like to migrate the team over at some point.
If I could have it my way, Dox site would be my #1 ask.
StuffbyYuki commentedon Jul 2, 2026
@c3us-dev I think we need to do some analysis on which tools use only the
manifest.jsonormanifest.json+ other dbt artifacts. And maybe we categorize them and decide to what to tackle first. We might be able to go pretty far just building themanifest.json.StuffbyYuki commentedon Jul 2, 2026
@krutoileshii I feel you! The docs feature is something we'll def do and will be a huge add. Since we have richer metadata, we can probably make it better than what dbt offers too.
krutoileshii commentedon Jul 2, 2026
I find docglow hi is pretty good for lineage and presentation, you could probably strip docglow down since sqlmesh has its own API that's much easier that. parsing manifests and stuff
[-]Export SQLMesh metadata as dbt-compatible artifacts `(manifest.json)` for ecosystem integration[/-][+]RFC: Export SQLMesh metadata as dbt-compatible artifacts `(manifest.json)` for ecosystem integration[/+]bosprimigenious commentedon Nov 3, 2026
Following the question about which consumers need more than
manifest.json, I checked two consumer loaders at pinned revisions:manifest.jsonandcatalog.json; it does not requestrun_results.json.Sources: OpenMetadata HTTP loader, optional object parsing, and dbt Docs bootstrap.
One useful distinction for the first acceptance fixture: OpenMetadata only enters its dbt-test processing branch when run-results are present (source). A manifest-only metadata fixture and an audit-result fixture therefore test different capabilities; the latter should use actual execution results rather than placeholder successes.
My suggestion is to validate manifest-only metadata ingestion first, then add a manifest+catalog docs fixture, keeping audit-results support separate. This is a source-level contract check, not a claim that a SQLMesh-generated manifest already passes either consumer's schema or integration. I have not run those integrations.
AI assistance: researched and checked with OpenAI Codex.