A Fine Slice Of SQL Server
Laura Graham-Brown makes use of a variable library:
You’ve set up your variable pipeline with WorkspaceIDs and LakehouseIDs etc etc Now you want to use that variable library in a data pipeline so we can use the common values and they will behave in a deployment pipeline.
Read on to see how it all works.
Leave a CommentJon Lunn checks the status of a data pipeline:
How do you check if a pipeline is running, not from the monitor, but from your Data Pipelines?
Maybe you’re like me and you have a Data Pipeline process that needs to check if some other pipeline else is running. In my case I have to check if a process is running due to Delta tables liking you to have one process writing to them, otherwise you can get concurrency issues as two items are trying to update the same delta table metadata file.
Those tricky metadata items like the process to be exclusive. It’s not just a Delta table issue; this can happen with regular SQL databases tables. So you can use this for anything you want to stop a locking issue or have an exclusive access to an object or just don’t want a process to run while another is doing its thing.
Read on to see how you can check the current status of a data pipeline from within a different data pipeline.
Leave a CommentMuhammad Hassan Arshad builds a pipeline:
The reality is that most companies generate data from multiple data sources. As data grows, one problem is consolidating data into a central repository for meaningful analysis and reporting. What options are available in the AWS ecosystem to achieve this?
Click through for a simple example.
Comments closedLaura Graham-Brown loads some data:
Dataverse is the data store behind parts of Dynamics and lots of Power Platform projects. So Dataverse can contain vital business data that will be needed for reporting. In this post we are going to look at one method which is using copy job with Dataverse to copy across data in Microsoft Fabric.
Click through to see how, including incremental data loads.
Comments closedIn Fabric Data Warehouse, this enhancement allows you to ingest, store, process, and analyze large descriptive text, logs, JSON, or spatial data—up to 16 MB per cell—without hitting size limits in the common warehouse scenarios.
Read on to see what’s included and what the current size limitations look like.
Comments closedReitse Eskens moves some data:
Well, unless you enjoy waking up every night to start your Extract-Transform-Load (ETL) process and manually running each process to do some work, it’s a smart move to automate this. Also, make sure everything always runs in the correct order. Additionally, there are situations where processes need to run in different configurations.
All these things can be done with what we call orchestration. It may sound a bit vague now, but we’ll get to the different moving parts of this, like parameterisation and pipelines.
Read on for a primer on the topic.
Comments closedAndy Brownsword doesn’t trust wizards, with their pointy caps and long beards:
If you need to create a copy of a table in another database, the ‘Import Data’ option may seem convenient. If you’ve used this method to copy to your dev environment and found things break, this post is for you.
Click through for some solid advice on how to import that data. Another thing I would sometimes do is coerce all of the input columns to long strings and load it into a staging table. Then, I could use T-SQL to re-shape the data however I needed it rather than trying to get a finicky SSIS flow to translate this date and time combination (or whatever) appropriately.
Comments closedJianlei Shen makes an announcement:
Copy Job Activity allows you to run Copy jobs as native activities inside Data Factory pipelines.
Copy jobs are created and managed independently in Data Factory for quick data movement between supported sources and destinations. With Copy job Activity, that same fast, lightweight experience is now embedded within pipelines, making it easier to automate, schedule, and chain Copy jobs as part of broader data workflows.
Read on for an overview of what’s in the activity and a few links on how to get started with it.
Comments closedAndy Brownsword saves some quid:
If you have a Self-Hosted Integration Runtime (SHIR, or IR for short here) on an Azure Virtual Machine (VM), there’s a cost to keep it online. When used intermittently – for example during batch processes – this is inefficient for costs as you’re paying for the compute you don’t need. One way to alleviate this is by controlling uptime of the environment manually, only bringing it online for as long as needed.
Read on to see how to do this.
Comments closedAndy Brownsword flips the script:
Last week we looked at calling a Data Factory Pipeline from a Logic App. This week I thought we’d balance it out by taking a look at calling a Logic App from an Azure Data Factory (ADF) Pipeline.
When building the Logic App last week we had to create our own polling mechanism to check for completion of the pipeline. The process is much simpler in the opposite direction. I specifically want to highlight two approaches, and save some pennies whilst we’re at it.
I am all about saving pennies, so be sure to check out that section as well.
Comments closed