A Fine Slice Of SQL Server
Michael Bourgon writes a script:
I hate creating SSIS. I also hate creating ADF via GUI. So here’s a way to do it with python.
There are a couple prereqs (below), specifically an ultra-configurable Linked Service and Dataset.
Click through for an example of how it works.
Leave a CommentPenny Zhou sees recent datasets:
How much time do you spend navigating to the same data sources when building dataflows? Data preparation is an iterative process—you often return to the same sources as you refine your dataflows, add new transformations, or create similar workflows. If you find yourself repeatedly connecting to the same tables, files, or databases, the Recent data module in Dataflow Gen2 is designed for you. This feature reduces friction by providing quick access to your most frequently used data items, letting you focus on the transformation logic rather than navigation.
Click through to see how you can access the Recent data menu and what it includes.
Leave a CommentKenneth Omorodion builds a Dataflows Gen2 pipeline:
In a previous tip, I described how we can implement snapshot reporting using Microsoft Fabric Dataflow Gen2. In this article, I will describe how to achieve the same using Microsoft Fabric Pipelines. I previously described how important snapshot reporting can be in Business Intelligence reporting. Some reasons why developers/engineers might prefer to leverage a Fabric pipeline instead of a Dataflow Gen 2 include considerations around cost efficiency and data volumes.
My strong preference is still to do this in code (notebooks, Spark jobs), but at least Dataflows Gen2 aren’t literally 100x slower than the alternatives anymore.
Leave a CommentKristina Mishra goes through a somewhat painful process:
Ah – you’ve finally got your Synapse Link set up and everything is trucking alone nicely. You’ve made it through Dev, QC, and Prod, and it’s been pretty hands off for a bit. Until, someone decides you need a refresh at your data source.
Click through for a story and several tips to reduce the pain level somewhat.
Leave a CommentLaura Graham-Brown shows another way to use variable libraries:
One of the popular low-code tools within Microsoft Fabric is the Gen2 Dataflow. Power BI report builders already know some Power Query. So armed with this knowledge is a popular starting point to load data into Microsoft Fabric. Adding values from the Variable Library in a Dataflow is an obvious plan to make it more future proof and to work better with Deployment pipelines.
I will confess the first time I tried these I could not get them to work till I read the instructions correctly. So they do work just understand the limitations!
To be fair, following instructions is one of the most challenging things to do, it seems.
Leave a CommentRebecca Lewis continues a series on Microsoft Fabric:
You already know the options. Run heavy reporting queries against production. eewgh. Or stand up a reporting replica, build ETL to keep it current, maintain a refresh schedule, and hope nothing breaks on a holiday weekend. It works, but it’s expensive and has an awful lot of moving pieces.
Fabric gives you a third path: continuously replicate your SQL Server data into OneLake using Fabric Mirroring, and let Power BI read it using Direct Lake mode. Your SQL Server stays focused on OLTP and your reporting runs against a near real-time copy in Fabric. No pipelines. No refresh schedules. Nice.
Read on for the options available with Microsoft Fabric, as well as an endearing note that “real-time” isn’t.
Leave a CommentLaura Graham-Brown makes use of a variable library:
You’ve set up your variable pipeline with WorkspaceIDs and LakehouseIDs etc etc Now you want to use that variable library in a data pipeline so we can use the common values and they will behave in a deployment pipeline.
Read on to see how it all works.
Comments closedJon Lunn checks the status of a data pipeline:
How do you check if a pipeline is running, not from the monitor, but from your Data Pipelines?
Maybe you’re like me and you have a Data Pipeline process that needs to check if some other pipeline else is running. In my case I have to check if a process is running due to Delta tables liking you to have one process writing to them, otherwise you can get concurrency issues as two items are trying to update the same delta table metadata file.
Those tricky metadata items like the process to be exclusive. It’s not just a Delta table issue; this can happen with regular SQL databases tables. So you can use this for anything you want to stop a locking issue or have an exclusive access to an object or just don’t want a process to run while another is doing its thing.
Read on to see how you can check the current status of a data pipeline from within a different data pipeline.
Comments closedMuhammad Hassan Arshad builds a pipeline:
The reality is that most companies generate data from multiple data sources. As data grows, one problem is consolidating data into a central repository for meaningful analysis and reporting. What options are available in the AWS ecosystem to achieve this?
Click through for a simple example.
Comments closedLaura Graham-Brown loads some data:
Dataverse is the data store behind parts of Dynamics and lots of Power Platform projects. So Dataverse can contain vital business data that will be needed for reporting. In this post we are going to look at one method which is using copy job with Dataverse to copy across data in Microsoft Fabric.
Click through to see how, including incremental data loads.
Comments closed