Category: ETL / ELT

Rebecca Lewis continues a series on Microsoft Fabric:

You already know the options. Run heavy reporting queries against production. eewgh. Or stand up a reporting replica, build ETL to keep it current, maintain a refresh schedule, and hope nothing breaks on a holiday weekend. It works, but it’s expensive and has an awful lot of moving pieces.

Fabric gives you a third path: continuously replicate your SQL Server data into OneLake using Fabric Mirroring, and let Power BI read it using Direct Lake mode. Your SQL Server stays focused on OLTP and your reporting runs against a near real-time copy in Fabric. No pipelines. No refresh schedules. Nice.

Read on for the options available with Microsoft Fabric, as well as an endearing note that “real-time” isn’t.

Leave a Comment

Laura Graham-Brown makes use of a variable library:

You’ve set up your variable pipeline with WorkspaceIDs and LakehouseIDs etc etc Now you want to use that variable library in a data pipeline so we can use the common values and they will behave in a deployment pipeline.

Read on to see how it all works.

Comments closed

Jon Lunn checks the status of a data pipeline:

How do you check if a pipeline is running, not from the monitor, but from your Data Pipelines?

Maybe you’re like me and you have a  Data Pipeline process that needs to check if some other pipeline else is running. In my case I have to check if a process is running due to Delta tables liking you to have one process writing to them, otherwise you can get concurrency issues as two items are trying to update the same delta table metadata file.

Those tricky metadata items like the process to be exclusive. It’s not just a Delta table issue; this can happen with regular SQL databases tables. So you can use this for anything you want to stop a locking issue or have an exclusive access to an object or just don’t want a process to run while another is doing its thing. 

Read on to see how you can check the current status of a data pipeline from within a different data pipeline.

Comments closed

Jovan Popovic has an update:

In Fabric Data Warehouse, this enhancement allows you to ingest, store, process, and analyze large descriptive text, logs, JSON, or spatial data—up to 16 MB per cell—without hitting size limits in the common warehouse scenarios.

Read on to see what’s included and what the current size limitations look like.

Comments closed

Reitse Eskens moves some data:

Well, unless you enjoy waking up every night to start your Extract-Transform-Load (ETL) process and manually running each process to do some work, it’s a smart move to automate this. Also, make sure everything always runs in the correct order. Additionally, there are situations where processes need to run in different configurations.

All these things can be done with what we call orchestration. It may sound a bit vague now, but we’ll get to the different moving parts of this, like parameterisation and pipelines.

Read on for a primer on the topic.

Comments closed