Press "Enter" to skip to content

Category: Spark

From SQL to PySpark and Spark SQL

Published 2026-09-03 by Kevin Feasel

Andy Brownsword gives Spark a try:

I’ve spent years shaping data with SQL Server, however after pulling at the threads of Fabric I’m opening notebooks and finding PySpark.

At first glance the difference is stark, but it’s not quite the dramatic shift it appears. If you’re not familiar, let’s look at what’s very similar, and where the true differences are.

There’s plenty of nuance in the syntax differences and behavioral differences between the platforms, but Spark SQL is just as ANSI compliant at this state as pretty much any other platform, and PySpark feels a lot like a chained quasi-functional approach to SQL because of Spark’s Scala heritage.

Leave a Comment

Spark SQL Temporary Views on Fabric Schema-Enabled Lakehouses

Published 2026-09-03 by Kevin Feasel

Gerhard Brueckl wants to create a temporary view:

Some time ago my friend Christian Henrik Reich blogged about how to handle schema-enabled lakehouses in Spark temporary views. We already found a good solution leveraging SQL USE keyword to set the schema once and reference tables by name only in our views. However, after some tests, in particular with notebookutils.notebook.runMultiple, I realized that there are some more things to consider as suddenly my SQL and notebooks stopped working when executed in parallel!

But let set the scope for this blogpost first. We recently built a data platform on Microsoft Fabric where we integrated data from different source systems which were combined into a single schema-enabled lakehouse where each source system had it own schema. Naturally, when querying those schemas, we used temporary views to express our business logic in SQL and then continue working with PySpark. We relied heavily on USE to work around the issue describe by Henrik until we realized that this approach does not work in combination with runMultiple. So we had to find another solution which I will describe here.

Read on for that solution.

Leave a Comment

Incremental Liquid Clustering in Apache Spark

Published 2026-07-28 by Kevin Feasel

Miles Cole lays out a recommendation:

Liquid Clustering was already a better abstraction than static partitioning due to its flexible nature. Fabric Spark Runtime 2.0 fixes the part that had me actively cautioning customers to reconsider blindly adopting it: the cost of maintaining the layout.

In Runtime 1.3 (Delta 3.2) a small append followed by OPTIMIZE would rewrite every file in a partial Z-Cube. In Runtime 2.0, the incremental strategy only touches files that are unclustered, small, or carrying a high density of deletion vectors. That changes Liquid Clustering from an occasional, potentially expensive maintenance operation into something that works beautifully with workloads of any shape and with adjacent layout optimizations. Batch or streaming writes. Auto Compaction and/or Fast Optimize. With the new incremental strategy, Liquid Clustering is highly compatible and highly efficient, and should now take its rightful place as the de facto new data layout strategy.

Read on to learn more about what Liquid Clustering is, how it works, and why you should think about it if you’re using Fabric Spark Runtime 2.0.

Comments closed

Saving Unity Catalog Tables in Microsoft OneLake

Published 2026-07-01 by Kevin Feasel

Gerhard Brueckl pushes boundaries:

Microsoft and Databricks recently announced the next step of their collaboration and integration. It is now possible to store Databricks Unity Catalog tables directly in Microsoft OneLake. Here are the official announcement from Microsoft: https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Extending-interoperability-Azure-Databricks-can-now-store-Unity/ba-p/5199741

Both parties have been working together to make this possible: Microsoft introduced the new item type Azure Databricks Storage and Databricks added support for OneLake for Unity Catalog External Locations (which can then be used to store the actual data). The UC External Location would then simply point to the storage endpoint provided by the Azure Databricks Storage item in Microsoft Fabric.

Click through to see what Gerhard found, as well as the results of some experimentation.

Comments closed

Spark DataFrameWriters

Published 2026-06-23 by Kevin Feasel

Miles Cole compares two generations of DataFrameWriter:

Most Spark developers learn to write data with df.write long before they ever encounter df.writeTo. It is simple, familiar, and everywhere: choose a format, pick a mode, add a few options, and save the result to a table or path. For years, that mental model worked well enough. Spark was often writing files first and tables second.

But modern lakehouse systems have changed the contract.

Read on to learn how, and what common problem the DataFrameWriterV2 is there to solve.

Comments closed

Identity Columns in Fabric Runtime 2.0

Published 2026-06-11 by Kevin Feasel

Jon Lunn is happy:

Fabric Runtime 2.0 is in public preview, and there is one small change for this that makes my life as a data engineer a little bit easier. Identity columns. I miss the good old T-SQL Identity column “Id INT IDENTITY(1,1)”, now we have something like it in Spark/Delta.

So with the the old spark/delta table runtime, you couldn’t have an default, automatically increasing column. But now we have these updates.

Click through to see how it works.

Comments closed

An Overview of the Fabric Native Execution Engine

Published 2026-02-27 by Kevin Feasel

Ankita Victor-Levi introduces a new processing model:

In today’s data landscape, as organizations scale their analytical workloads, the demand for faster, more cost-efficient computation continues to rise. Apache Spark has long been the backbone of largescale data processing with its in‑memory processing and powerful APIs, but today’s workloads demand even better performance.

Microsoft Fabric addresses this challenge with the Native Execution Engine—a vectorized, C++ powered execution layer that accelerates Spark jobs with no code changes, reduced runtime, and at no additional compute cost. This blog post will take you behind the scenes to give an overview of how the engine works and how it delivers performance gains while preserving the familiar Spark developer experience users already know and love.

Read on to learn more about its capabilities and current limitations.

Comments closed

Spark Schema Inference in Production

Published 2026-02-23 by Kevin Feasel

Miles Cole shares some advice:

To show the impact I want to highlight a benchmark that included Fabric Spark on a single 19GB CSV input file (100M Contoso dataset, sales table) for the benchmark. While there were a number of issue with this benchmark that inadvertently make Spark appear to be slow, this is only focused on the impact of inferring schema and practical recommendations.

Read on to see a performance problem that schema inference brings up. I’d also want to mention the risk of data updates blowing up your well-laid plans as a risk. Schema inference is a double-edged sword: it can be convenient and open up new approaches to development, but can just as easily cause unexpected failures.

Comments closed

Creating a Spark Job Definition

Published 2026-02-06 by Kevin Feasel

Miles Cole builds a job:

A Spark Job Definition is effectively a way to run a packaged Spark application, Fabric’s version of executing a spark-submit job. You define:

  • what code should run (the entry point),
  • which files or resources should be shipped with it,
  • and which command-line arguments should control its behavior.

Unlike a notebook, there is no interactive editor or cell output, but this is arguably not a missing feature, it’s the whole point… an SJD is not meant for exploration; it is meant to deterministically run a Spark application.

With that concept in mind, click through for the process.

Comments closed

Combining Fabric Real-Time Intelligence, Notebooks, and Spark Structured Streaming

Published 2026-02-06 by Kevin Feasel

Arindam Chatterjee and QiXiao Wang show off some preview functionality:

Building event-driven, real-time applications using Fabric Eventstreams and Spark Notebooks just got a whole lot easier. With the Preview of Spark Notebooks and Real-Time Intelligence integration — a new capability that brings together the open-source community supported richness of Spark Structured Streaming with the real-time stream processing power of Fabric Eventstreams — developers can now build low-latency, end-to-end real-time analytics and AI pipelines all within Microsoft Fabric.

You can now seamlessly access streaming data from Eventstreams directly inside Spark notebooks, enabling real-time insights and decision-making without the complexity & tediousness of manual coding and configuration.

Click through to learn more.

Comments closed