Press "Enter" to skip to content

Category: Microsoft Fabric

Finding Mutable and Immutable Properties in Microsoft Fabric Spark

Published 2024-11-01 by Kevin Feasel

Sandeep Pawar wants to make a change:

Spark properties are divided into mutable and immutable configurations based on whether they can be safely modified during runtime after the spark session is created.

Mutable properties can be changed dynamically using spark.conf.set() without requiring a restart of the Spark application – these typically include performance tuning parameters like shuffle partitions, broadcast thresholds, AQE etc.

Immutable properties, on the other hand, are global configurations that affect core spark behavior and cluster setup and these must be set before/at session initialization as they require a fresh session to take effect.

Read on to see how you can tell which is which.

Leave a Comment

T-SQL Notebooks in Microsoft Fabric

Published 2024-10-31 by Kevin Feasel

Dennes Torres tries out T-SQL notebooks:

T-SQL Notebooks is one of the new features announced during FabCon Europe.

The most distracted could miss the fact this is a new feature at all. Yes, it is. Notebooks were capable to support Spark SQL, but T-SQL is something new.

The main examples being announced are built with data warehouses, but let me confirm and highlight this:

T-SQL Notebooks support lakehouses as well.

There is at least one limitation: DML is not supported with lakehouses.

Saving my rant about lakehouses vs warehouses in Fabric, do read what Dennes has to say about T-SQL notebooks as they exist today.

Comments closed

Dynamically Running Notebooks across Fabric Lakehouse Environments

Published 2024-10-30 by Kevin Feasel

Ayman El-Ghazali solves a problem:

A few months ago, an ISV customer approached with a request to have notebooks run across Microsoft Fabric Lakehouse environments dynamically.  Initially the first request was to allow pipelines in Fabric to pass parameters for file paths to help with data ingestion.  This would allow the customer to use the same notebook across Lakehouse environments for the customers that they are serving. After resolving this, the scope increased to include the notebook execution. The notebooks should be able to run across workspace environments and not have to be attached to a Lakehouse at the time of execution.  The solution presented below allows for the customer to run notebooks across environments but also allows them to run SQL queries against existing Lakehouse tables; additionally it allows for access to tables created during the notebook execution run without the notebook being attached to the Lakehouse. 

Read on to learn how.

Comments closed

KQLMagic in Fabric Runtime 1.3

Published 2024-10-30 by Kevin Feasel

Sandeep Pawar spreads the news:

I wrote a blog last year on the usefulness of KQLMagic command in Fabric notebook and made a suggestion that it should be part of the default runtime. Well, guess what – it’s now in the Fabric Runtime 1.3. No installation necessary and authentication is handled automatically.

Read on to learn more about how you can use KQLMagic in a Microsoft Fabric notebook to read from an Eventhouse.

Comments closed

Viewing Storage Consumption in Microsoft Fabric

Published 2024-10-30 by Kevin Feasel

Gilbert Quevauvilliers wants to know about storage utilization in Microsoft Fabric:

This blog post will show you how to understand what is consuming your Fabric Storage.

If you want to know how I got this data, please read my previous blog post View all your Storage consumed in Microsoft Fabric – Lakehouse Files, Tables and Warehouses – FourMoo

With this Semantic model below, I could also create alerts to notify based on certain thresholds. For example, if total storage in a single App workspace is more than 100GB send me an alert (This could be done using Power Automate). Or it could be on too many files being stored, or even looking at the Parquet file sizes and if they are too small they would then need to be optimised (for better performance).

Click through for the report.

Comments closed

Updating Microsoft Fabric Warehouses via Power Apps

Published 2024-10-28 by Kevin Feasel

Shabnam Watson troubleshoots an issue:

One of my recent explorations with Microsoft Fabric was integrating Power Apps with a Fabric Warehouse—both in a standalone Power Apps app and as an embedded visual within a Power BI report to enable writeback. My goal was simple: to enable Power Apps to display and update records from a table in the Fabric Warehouse. Initially, I turned to the three-screen template apps to get started, however; while it displayed the records, it failed to update them. This led me to dive deeper into how Fabric Warehouse differs from other SQL data sources when it comes to Power Apps and to find a workaround.

Read on to see how it all works.

Comments closed

Moving items from “My Workspace” in Microsoft Fabric

Published 2024-10-28 by Kevin Feasel

Matt Collins is on the move:

A common issue I’ve seen recently when working with Microsoft Fabric is managing items in the “My Workspace” Workspace. This is often the playground for many users who sign up for a free trial but can result in some administrative overhead when resources developed here are now ready for wider use and need to be moved to a shared location.

In this article we will discuss how to move workspace items in Microsoft Fabric from “My Workspace” to other workspaces, using our understanding of item dependencies and some metadata to speed up the process.

Read on to learn how, as well as some of the issues you can run into along the way.

Comments closed

The Importance of Monitoring in Microsoft Fabric

Published 2024-10-25 by Kevin Feasel

Marc Lelijveld flips a switch but also watches it:

A long time ago, I blogged about Power BI governance with topics like feature implementation in a phased approach and why you should consider to disable export to Excel. In this blog, I want to continue the governance topic with another blog about why monitoring your tenant is important! This blog will also provide you an overview of the various monitoring options you have out of the box, no matter what your role is. No matter if you are the workspace-, capacity-, domain- or tenant administrator.

I encourage everyone, no matter if you are the service administrator or not, to go through this blog and look from various angles how monitoring can help. I think it can be relevant for any Fabric / Power BI user to see all capabilities it has to offer from a different angle and better understand possible restrictions that are set by your service administrator.

Read on for Marc’s argument, as well as plenty of examples of what you can do as far as monitoring goes.

Comments closed

Dynamically Start a Collection of Child Pipelines in Fabric Data Factory

Published 2024-10-24 by Kevin Feasel

Andy Leonard continues a series on Microsoft Fabric Data Factory:

In this post, I modify the dynamic parent pipeline from the previous post to explore calling several child pipelines that may be called by a parent pipeline. In this post, we will:

  • Clone the child pipeline (twice)
  • Copy the cloned child pipeline id values
  • Clone the dynamic parent pipeline from the previous post
  • Add and configure a pipeline variable for an array of child pipeline ids
  • Add and configure a ForEach
    • Move the “Invoke Pipeline (Preview)” activity
    • Configure the “ForEach”
    • Configure the “Invoke Pipeline (Preview)” Activity to Use “ForEach” Items
  • Test the execution of a dynamic collection of child pipelines

Andy’s got quite a bit in this post, so check it out.

Comments closed

A Primer on Medallion Architecture in Microsoft Fabric

Published 2024-10-24 by Kevin Feasel

Kenneth Omorodion builds a warehouse:

Data warehouses are essential components of modern analytics systems, offering optimized storage and processing capabilities for large volumes of data. When integrated with a Lakehouse architecture, you can combine the best of both worlds—structured, schema-enforced data storage with the flexibility and scalability of data lakes. Microsoft Fabric provides an excellent environment for implementing the Medallion Architecture, a design pattern for building efficient data processing pipelines by layering data into bronze, silver, and gold zones.

Click through for the process.

Comments closed