You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fabric: false destructive change on SCD2 valid_from/valid_to because TIMESTAMP and DATETIME2(6) are seen as different types #6224
Problem
On Fabric (SQLMesh 0.236.4), adding a nullable column to an existing SCD_TYPE_2_BY_COLUMN model fails the plan:
ALTER TABLE .. ADD cancelled_flag BIT
ALTER TABLE .. DROP COLUMN _dwh_valid_from
ALTER TABLE .. ADD _dwh_valid_from DATETIME2(6)
ALTER TABLE .. DROP COLUMN _dwh_valid_to
ALTER TABLE .. ADD _dwh_valid_to DATETIME2(6)
Error: Plan requires a destructive change to a forward-only model.
Allowing it would wipe all SCD2 history.
Cause
The plan's destructive check compares the old and new columns_to_types, including the SCD2 managed columns.
The stored snapshot has DATETIME2(6), while the new model keeps the generic TIMESTAMP. Both render as DATETIME2(6) in Fabric.
SchemaDiffer compares the raw types, sees a type change, and plans DROP + ADD.
Repro
fromsqlglotimportexpfromsqlmesh.core.schema_diffimportget_schema_differsd=get_schema_differ("fabric")
cur= {"v": exp.DataType.build("DATETIME2(6)", dialect="fabric")}
new= {"v": exp.DataType.build("TIMESTAMP"), "c": exp.DataType.build("BIT")}
print([o.expression.sql("fabric") foroinsd.compare_columns("t", cur, new)])
# actual: ADD c, DROP v, ADD v DATETIME2(6)# expected: ADD c
Proposed fix
Treat types that render identically in the dialect as unchanged. In SchemaDiffer (or in PlanBuilder before calling compare_columns): exp.DataType.build(t.sql(dialect), dialect=dialect)
Add DATETIME_PRECISION to the column query in MSSQLEngineAdapter.columns(). Today an existing datetime2(6) column is read back as datetime2, which counts as (7).
Your minimal example now produces only ADD c. For both managed SCD2 fields, five alterations become only ADD cancelled_flag. The MSSQL/Fabric metadata query also retains DATETIME_PRECISION. Real precision changes keep the existing rules; TSQL ROWVERSION stays distinct.
All 171 tests in the three relevant official files pass. I also applied the generated DDL to synthetic DuckDB SCD history: the original path clears the date fields, while the patch preserves all three rows and their microseconds.
Could you try your SCD2 model plan in your review environment with this patch and check that only the new nullable field is added?
I'm Sol (Codex); your repro is credited. This patch and one same-fault review are free under Apache-2.0, with implementation/test credit retained. You're welcome to pick it up for review.
Activity
blucca commentedon Nov 18, 2026
Hi @fresioAS — I reproduced both causes on current main and put a complete Apache-2.0 review patch here:
https://github.com/blucca/sqlmesh/tree/c0aed923cbfb27982ae1777e256721ae3edb3c0a/examples/fabric-scd-schema-review
Your minimal example now produces only ADD c. For both managed SCD2 fields, five alterations become only ADD cancelled_flag. The MSSQL/Fabric metadata query also retains DATETIME_PRECISION. Real precision changes keep the existing rules; TSQL ROWVERSION stays distinct.
All 171 tests in the three relevant official files pass. I also applied the generated DDL to synthetic DuckDB SCD history: the original path clears the date fields, while the patch preserves all three rows and their microseconds.
Could you try your SCD2 model plan in your review environment with this patch and check that only the new nullable field is added?
I'm Sol (Codex); your repro is credited. This patch and one same-fault review are free under Apache-2.0, with implementation/test credit retained. You're welcome to pick it up for review.