Press "Enter" to skip to content

Category: Spark

Saving Unity Catalog Tables in Microsoft OneLake

Published 2026-07-01 by Kevin Feasel

Gerhard Brueckl pushes boundaries:

Microsoft and Databricks recently announced the next step of their collaboration and integration. It is now possible to store Databricks Unity Catalog tables directly in Microsoft OneLake. Here are the official announcement from Microsoft: https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Extending-interoperability-Azure-Databricks-can-now-store-Unity/ba-p/5199741

Both parties have been working together to make this possible: Microsoft introduced the new item type Azure Databricks Storage and Databricks added support for OneLake for Unity Catalog External Locations (which can then be used to store the actual data). The UC External Location would then simply point to the storage endpoint provided by the Azure Databricks Storage item in Microsoft Fabric.

Click through to see what Gerhard found, as well as the results of some experimentation.

Leave a Comment

Spark DataFrameWriters

Published 2026-06-23 by Kevin Feasel

Miles Cole compares two generations of DataFrameWriter:

Most Spark developers learn to write data with df.write long before they ever encounter df.writeTo. It is simple, familiar, and everywhere: choose a format, pick a mode, add a few options, and save the result to a table or path. For years, that mental model worked well enough. Spark was often writing files first and tables second.

But modern lakehouse systems have changed the contract.

Read on to learn how, and what common problem the DataFrameWriterV2 is there to solve.

Leave a Comment

Identity Columns in Fabric Runtime 2.0

Published 2026-06-11 by Kevin Feasel

Jon Lunn is happy:

Fabric Runtime 2.0 is in public preview, and there is one small change for this that makes my life as a data engineer a little bit easier. Identity columns. I miss the good old T-SQL Identity column “Id INT IDENTITY(1,1)”, now we have something like it in Spark/Delta.

So with the the old spark/delta table runtime, you couldn’t have an default, automatically increasing column. But now we have these updates.

Click through to see how it works.

Comments closed

An Overview of the Fabric Native Execution Engine

Published 2026-02-27 by Kevin Feasel

Ankita Victor-Levi introduces a new processing model:

In today’s data landscape, as organizations scale their analytical workloads, the demand for faster, more cost-efficient computation continues to rise. Apache Spark has long been the backbone of largescale data processing with its in‑memory processing and powerful APIs, but today’s workloads demand even better performance.

Microsoft Fabric addresses this challenge with the Native Execution Engine—a vectorized, C++ powered execution layer that accelerates Spark jobs with no code changes, reduced runtime, and at no additional compute cost. This blog post will take you behind the scenes to give an overview of how the engine works and how it delivers performance gains while preserving the familiar Spark developer experience users already know and love.

Read on to learn more about its capabilities and current limitations.

Comments closed

Spark Schema Inference in Production

Published 2026-02-23 by Kevin Feasel

Miles Cole shares some advice:

To show the impact I want to highlight a benchmark that included Fabric Spark on a single 19GB CSV input file (100M Contoso dataset, sales table) for the benchmark. While there were a number of issue with this benchmark that inadvertently make Spark appear to be slow, this is only focused on the impact of inferring schema and practical recommendations.

Read on to see a performance problem that schema inference brings up. I’d also want to mention the risk of data updates blowing up your well-laid plans as a risk. Schema inference is a double-edged sword: it can be convenient and open up new approaches to development, but can just as easily cause unexpected failures.

Comments closed

Creating a Spark Job Definition

Published 2026-02-06 by Kevin Feasel

Miles Cole builds a job:

A Spark Job Definition is effectively a way to run a packaged Spark application, Fabric’s version of executing a spark-submit job. You define:

  • what code should run (the entry point),
  • which files or resources should be shipped with it,
  • and which command-line arguments should control its behavior.

Unlike a notebook, there is no interactive editor or cell output, but this is arguably not a missing feature, it’s the whole point… an SJD is not meant for exploration; it is meant to deterministically run a Spark application.

With that concept in mind, click through for the process.

Comments closed

Combining Fabric Real-Time Intelligence, Notebooks, and Spark Structured Streaming

Published 2026-02-06 by Kevin Feasel

Arindam Chatterjee and QiXiao Wang show off some preview functionality:

Building event-driven, real-time applications using Fabric Eventstreams and Spark Notebooks just got a whole lot easier. With the Preview of Spark Notebooks and Real-Time Intelligence integration — a new capability that brings together the open-source community supported richness of Spark Structured Streaming with the real-time stream processing power of Fabric Eventstreams — developers can now build low-latency, end-to-end real-time analytics and AI pipelines all within Microsoft Fabric.

You can now seamlessly access streaming data from Eventstreams directly inside Spark notebooks, enabling real-time insights and decision-making without the complexity & tediousness of manual coding and configuration.

Click through to learn more.

Comments closed

More Spark Jobs, Fewer Notebooks

Published 2026-02-05 by Kevin Feasel

Miles Cole lays out an argument:

I’m guilty. I’ve peddled the #NotebookEverything tagline more than a few times.

To be fair, notebooks are an amazing entry point to coding, documentation, and exploration. But this post is dedicated to convincing you that notebooks are not, in fact, everything, and that many production Spark workloads would be better executed as a non-interactive Spark Job.

Miles has a “controversial claim” at the end that I don’t think is particularly controversial at all. I agree with pretty much the entire article, especially around the difficulties of testing notebooks properly.

Comments closed

Efficient Sampling of Spark Datasets

Published 2026-01-23 by Kevin Feasel

Rajesh Vakkalagadda needs a sample:

Sampling is a fundamental process in machine learning that involves selecting a subset of data from a larger dataset. This technique is used to make training and evaluation more efficient, especially when working with massive datasets where processing every data point is impractical

However, sampling comes with its own challenges. Ensuring that samples are representative is crucial to prevent biases that could lead to poor model generalization and inaccurate evaluation results. The sample size must strike a balance between performance and resource constraints. Additionally, sampling strategies need to account for factors such as class imbalance, temporal dependencies, and other dataset-specific characteristics to maintain data integrity.

Click through for an answer in Scala. The Python implementation would be very similar,

Comments closed

Writing Sparse Pandas DataFrames to S3

Published 2026-01-19 by Kevin Feasel

Pooja Chhabra tries a few things:

If you’ve worked with large-scale machine learning pipelines, you must know one of the most frustrating bottlenecks isn’t always found in the complexity of the model or the elegance of the architecture — it’s writing the output efficiently.

Recently, I found myself navigating a complex data engineering hurdle where I needed to write a massive Pandas sparse DataFrame — the high-dimensional output of a CountVectorizer — directly to Amazon S3. By massive, I mean tens of gigabytes of feature data stored in a memory-efficient sparse format that needed to be materialized as a raw CSV file. This legacy requirement existed because our downstream machine learning model was specifically built to ingest only that format, leaving us with a significant I/O challenge that threatened to derail our entire processing timeline.

Read on for two major constraints, a variety of false starts, and what eventually worked.

Comments closed