Kenneth Omorodion builds a Dataflows Gen2 pipeline:
In a previous tip, I described how we can implement snapshot reporting using Microsoft Fabric Dataflow Gen2. In this article, I will describe how to achieve the same using Microsoft Fabric Pipelines. I previously described how important snapshot reporting can be in Business Intelligence reporting. Some reasons why developers/engineers might prefer to leverage a Fabric pipeline instead of a Dataflow Gen 2 include considerations around cost efficiency and data volumes.
My strong preference is still to do this in code (notebooks, Spark jobs), but at least Dataflows Gen2 aren’t literally 100x slower than the alternatives anymore.