A Fine Slice Of SQL Server
Brian Bønk does some data shuffling:
The Fabric servive Eventstream can read data from an Event Hub service to help collect data from IoT devices and other streaming services. But how do you configure the Event Hub and Eventstream to work together?
I’ll try to help with that in this blogpost.
Click through for the article, which provides a how-to guide on the topic.
Comments closedPaul White loads things quickly:
All this can be achieved with client-side tools and programming. It can also be done server-side by importing the raw data into a staging table before processing using T-SQL procedures.
Other times, the need arises to ingest data without using client-side tools and without making a complete copy of the raw data on the server. This article describes one possible approach in that situation.
Read on for the process.
Comments closedReitse Eskens uploads some data:
In a blog from a few weeks ago, I wrote about getting data from your on-prem SQL Server into Fabric. At the time, the only option for a copy dataflow was using a direct connection over the internet. It still is, but now you can also use the PowerBI Data Gateway to get data from your SQL Server into Fabric.
In this blog, I’ll take you through the steps needed and an issue I ran into.
Read on for Reitse’s instructions and how to avoid the issue he ran into.
Comments closedEitan Blumin handles a migration:
A few months ago, I was involved in an interesting project where a large customer (not to be named due to NDA) needed to migrate their entire Azure cloud subscription to another subscription. This was a difficult and arduous process that involved several PaaS technologies, besides SQL Server, that I didn’t have experience with before.
But it presented very interesting challenges and opportunities to learn new things.
One of these was the need to migrate an entire Azure Cosmos DB with Table Storage API account from one subscription to another.
Read on for the challenge, the intermediate solution using the Cosmos DB Data Migration Tool, and Eitan’s Powershell script to automate the process. I know and work with most of the people working on the DMT and they’re good folks.
Comments closedChen Hirsh looks at some updates:
A few months ago I wrote a post about the new feature of change data capture (CDC) on Azure data factory (ADF) – https://www.madeiradata.com/post/the-wind-of-change-change-data-capture-in-data-factory
Change data capture, as the name suggests, gets the data changes on one system, and replicates them to another. Since this is a task that data engineers do a lot, this was a very welcome addition to ADF.
In this post, we’ll explore what is new on this front.
Click through for what’s new, though do be cognizant of which items are in GA and which are still in preview.
Comments closedIn Azure data factory (ADF), An integration runtime is a compute resource to run your pipelines on. When you run an application on your computer, it uses the computer resources, such as CPU and memory, to run its tasks. When you run activities in a pipeline in ADF, they also need resources to do their job, like copying data or writing a file, and these are provided by the integration runtime.
When you create an instance of ADF, you get a default integration runtime, hosted in the same region that you created ADF in. If you need, you can add your own integration runtimes, either on Azure, or you can download and install a self-hosted integration runtime (SHIR) on your own server.
Read on to understand when you would want to use a self-hosted integration runtime and the process to do so. This SHIR also applies to Synapse pipelines and is one of the few ways to move data out of a Synapse workspace with data exfiltration protection enabled.
Comments closedTeo Lachev reviews the primary methods for data ingestion in Microsoft Fabric:
Fabric supports three options for automated data integration: Data Pipeline (Azure Data Factory pipeline), Dataflow Gen2 (Power BI dataflow), and Notebook (Spark). I summarize these three options in the following table, which loosely resembles the Microsoft comparison table but with my take on it.
Read on for Teo’s thoughts on the matter.
Comments closedAndy Leonard troubleshoots an error:
Have you encountered this error: “Failed Get Parameter Info of parameter ____”? I ran into this error a couple times while consulting for a client to lift and shift SQL Server Integration Services (SSIS) to Azure Data Factory (ADF).
Click through for an example and what is the most likely cause of the problem.
Comments closedDid you know Azure Data Factory (ADF) will actually list available properties? It will. One of the things I cover in my ADF training titled Master the Fundamentals of Azure Data Factory is this handy troubleshooting tip.
Read on to see how, though I’d personally like something which is a bit faster than waiting for the thing to execute and getting back what my choices are.
Comments closedReitse Eskens spends an hour or so:
In my previous blogs, I’ve written about Fabric and all the cool things it can do. Thing is, my load tests were based on files. Either CSV or Delta. But in reality, a lot of data comes from an on-premises database server. In reality, you might connect to a SQL 2008 instance or maybe even older. Truth be told, I haven’t got an instance in that version/edition around anymore. So I had to use SQL Server 2019, a version I’m encountering more often nowadays.
For this blog, it won’t make much sense to create a humongous database and try to get all the data in. Fabric will cope, the major issue (in my experience) is the internet connection between my local database and the Fabric environment. One thing I’m really curious about is if Fabric will have the Link capability that was introduced for Synapse Analytics and SQL Server 2022.
There’s no Link capability currently available, so Reitse does the next-best thing and uses Fabric pipelines.
Comments closed