A Fine Slice Of SQL Server
Kristina Mishra goes through a somewhat painful process:
Ah – you’ve finally got your Synapse Link set up and everything is trucking alone nicely. You’ve made it through Dev, QC, and Prod, and it’s been pretty hands off for a bit. Until, someone decides you need a refresh at your data source.
Click through for a story and several tips to reduce the pain level somewhat.
Comments closedDevang Shah and Slava Trofimov show off a design pattern:
This design pattern provides intuitive, interactive Fabric-native experiences for any user:
- Intelligent time binning: Handle billions of data points by automatically grouping them into optimal intervals.
- Time brushing: Zoom in any period with drag-and-select interactions.
- Multi-metric comparison: View multiple time series side by side across different assets.
- Flexible aggregation: Switch between average, min, max, and sum with a single selection.
- Anomaly detection: KQL queries detect unusual patterns in your time series with no ML expertise required.
- Statistical insights: View descriptive statistics and correlations.
- Contextualization: Bring asset hierarchies, tag metadata, and definitions directly into the report for richer interpretation.
Read on to learn more about the pattern and how it works. There are a lot of moving parts to get right, but the end result looks impressive.
Comments closedLaura Graham-Brown shows another way to use variable libraries:
One of the popular low-code tools within Microsoft Fabric is the Gen2 Dataflow. Power BI report builders already know some Power Query. So armed with this knowledge is a popular starting point to load data into Microsoft Fabric. Adding values from the Variable Library in a Dataflow is an obvious plan to make it more future proof and to work better with Deployment pipelines.
I will confess the first time I tried these I could not get them to work till I read the instructions correctly. So they do work just understand the limitations!
To be fair, following instructions is one of the most challenging things to do, it seems.
Comments closedRebecca Lewis continues a series on Microsoft Fabric:
You already know the options. Run heavy reporting queries against production. eewgh. Or stand up a reporting replica, build ETL to keep it current, maintain a refresh schedule, and hope nothing breaks on a holiday weekend. It works, but it’s expensive and has an awful lot of moving pieces.
Fabric gives you a third path: continuously replicate your SQL Server data into OneLake using Fabric Mirroring, and let Power BI read it using Direct Lake mode. Your SQL Server stays focused on OLTP and your reporting runs against a near real-time copy in Fabric. No pipelines. No refresh schedules. Nice.
Read on for the options available with Microsoft Fabric, as well as an endearing note that “real-time” isn’t.
Comments closedAaron Merrill introduces a new whitepaper:
In our whitepaper, The future of data security is interoperability, we make the case for a different data foundation: interoperable security that’s defined once and enforced everywhere your data is used. Using OneLake security as the lens, it walks through the core concepts and architectural choices behind centralized policy definition with distributed, engine-level enforcement, and explores how fine-grained access controls and enterprise governance fit into a multi-engine world.
Click through for Aaron’s summary and check out the link for the whitepaper itself, in PDF format.
Comments closedPaul Hernandez builds a (virtual) network:
Mirroring has been a transformative technology for data integrations tasks since the early Microsoft Fabric days. Moreover, this feature has been called “pain killer as a service” in community posts. In many projects, data sources to be mirrored are behind private networking and for security reasons they are not accessible using public internet. If you want to mirror, for example, an Azure SQL database, you’ll need a data gateway. According to the official docs: “If your Azure SQL Database is not publicly accessible and doesn’t allow Azure services to connect to it, you can set up virtual network data gateway or on-premises data gateway to mirror the data”.
In this post I’ll show you step-by-step how to set up connectivity to be able to use mirroring when Azure SQL allows only private access.
There are several steps involved, but the end result is worth it compared to not having the data at all or needing to make it accessible over the Internet.
Comments closedGilbert Quevauvilliers backs into a number:
I was recently working with a customer and one of the questions they had is we are going to be running an ingestion process. We want to know how much Fabric Capacity this will be consuming.
The challenge with this question is that in Fabric a background capacity gets smoothed over 24 hours.
For example, when looking at the Capacity Metrics App I can see my overall usage, but HOW MUCH CAPACITY IS IT CONSUMING?
Read on for the answer.
Comments closedArindam Chatterjee and QiXiao Wang show off some preview functionality:
Building event-driven, real-time applications using Fabric Eventstreams and Spark Notebooks just got a whole lot easier. With the Preview of Spark Notebooks and Real-Time Intelligence integration — a new capability that brings together the open-source community supported richness of Spark Structured Streaming with the real-time stream processing power of Fabric Eventstreams — developers can now build low-latency, end-to-end real-time analytics and AI pipelines all within Microsoft Fabric.
You can now seamlessly access streaming data from Eventstreams directly inside Spark notebooks, enabling real-time insights and decision-making without the complexity & tediousness of manual coding and configuration.
Click through to learn more.
Comments closedRob Sewell walks through a recent experience:
Firstly, an apology to my friends (especially Randolph) in the documentation team at Microsoft. I know how hard you work to produce accurate and useful documentation, and I appreciate your efforts. This is not a criticism of your work, but rather an observation about the challenges I faced.
This is a story about a recent experience and the lessons learned along the way.
Read on for the issue and what Rob had to do. This is a case study in how hard it is to write good documentation, especially around the edges of what is possible.
Comments closedMichal Bar shows off a new capability:
A frequent request we receive from dashboard authors is the ability to have greater control over color settings.
Until now, color assignments in real-time dashboards were largely automatic. While this worked for basic scenarios, it often fell short in operational and reporting use cases where color isn’t decoration—it’s meaning. Data Series Colors is a new capability that gives authors direct control over how colors are applied to their visuals.
Read on to see how it works for real-time dashboards.
Comments closed