Press "Enter" to skip to content

Category: Performance Tuning

Incremental Liquid Clustering in Apache Spark

Miles Cole lays out a recommendation:

Liquid Clustering was already a better abstraction than static partitioning due to its flexible nature. Fabric Spark Runtime 2.0 fixes the part that had me actively cautioning customers to reconsider blindly adopting it: the cost of maintaining the layout.

In Runtime 1.3 (Delta 3.2) a small append followed by OPTIMIZE would rewrite every file in a partial Z-Cube. In Runtime 2.0, the incremental strategy only touches files that are unclustered, small, or carrying a high density of deletion vectors. That changes Liquid Clustering from an occasional, potentially expensive maintenance operation into something that works beautifully with workloads of any shape and with adjacent layout optimizations. Batch or streaming writes. Auto Compaction and/or Fast Optimize. With the new incremental strategy, Liquid Clustering is highly compatible and highly efficient, and should now take its rightful place as the de facto new data layout strategy.

Read on to learn more about what Liquid Clustering is, how it works, and why you should think about it if you’re using Fabric Spark Runtime 2.0.

Comments closed

Improving Power BI Performance

James Serra has updated a guide on Power BI performance optimizations:

This post is a complete replacement for and update to my April 2022 post, Power BI Performance Features. That post focused on several performance features that were new or in preview at the time. Four years later, many of those features have matured, Microsoft Fabric has changed the architecture choices available to us, and Direct Lake has become an important part of the conversation. So rather than keep adding notes to an old feature list, I decided to step back and answer the more useful question: How should you design, diagnose, and improve Power BI performance today?

Here’s the bottom line: Power BI performance is not one feature, one setting, or one capacity size. It is the result of a chain that includes the data source, data preparation, storage mode, semantic model, DAX, report design, security, refresh process, and Fabric capacity. A weakness anywhere in that chain can become the bottleneck, which is why randomly changing DAX or buying more capacity often does not solve the real problem. The best approach is to make the right architectural choices first, measure what is slow, and then fix the layer that is actually causing the delay. For more info, see Optimization guide for Power BI.

Click through for James’s guidance.

Comments closed

Performance Tuning with Microsoft Fabric Warehouse Query Insights

Koen Verbeeck troubleshoots query performance:

We are in the process of building a data warehouse in Microsoft Fabric. Since we have good knowledge of T-SQL, we are using Fabric Warehouse as the database engine. We are hitting some performance issues in our ETL and we would like to investigate further, but traditional SQL Server DMVs don’t give the results we are hoping for. How can we troubleshoot performance in the Fabric Warehouse?

Click through to see how.

Comments closed

Dealing with linger.ms in Apache Kafka

Jack Vanlightly covers a common performance optimization:

Recently I was curious to see if there was any general performance improvement since Kafka 3.X. So I ran a suite of benchmarks with Dimster against 3.7.2 and 4.3.0. I saw two common patterns:

Those two patterns involved higher latency for the newer version of Kafka, but better scale. Click through to understand what changed between these two versions that had such a big impact.

Comments closed

Share Groups and Sub-Optimal Performance

Jack Vanlightly creates a problem:

In this post we’re going to see how share.acquire.mode=record_limit combined with:

  • fewer consumers than partitions
  • and various cases of “partition skew”

…can result in subpar performance with share groups. 

I stumbled on these issues when running large sets of dimensional tests with Dimster’s explore-limits mode, which finds the highest sustainable throughput while staying within a target end-to-end latency target. There was a specific subset of the tests that explore-limits mode would consistently fail to complete, and they all happened to be with record_limit and a consumer count lower than the partition count. In this test, we’ll understand why Dimster had such a hard time with this combination.

Click through for the details, as well as how to mitigate this sort of scenario.

Comments closed

The Basics of Query Folding with Power BI

Andy Brownsword explains one performance improvement technique for Power BI data transformations:

As a database developer, when I started using Power BI, I was concerned about it retrieving reams of data only to perform transformations downstream. The Power Query editor misleads us into thinking the retrieval and transformations are applied sequentially.

Thanks to Query Folding, that’s not usually the case. And that gives us more power to extract performance from the database.

This only works in situations where there’s something downstream to perform that processing, like a relational database. And one of the areas where you can affect performance, either positively or negatively, is in organizing operations such that you have a stretch of foldable operations. That way, all of it can run as one operation in the database.

Comments closed

An Introduction to Memory Grants in SQL Server

Erik Darling has a new video for us:

Erik monitoring tool mogul darling here with Darling Data. In today’s video, much like I think I foreshadowed in yesterday’s office hours video, we are going to talk about memory grants. We’re going to do a somewhat gentle introduction to them and then in the next video we’ll talk a little bit more about where they get interesting. 

Erik shares his perfectly reasonable take on the nature of strings. I’d probably also get rid of dates as well—too much confusion there for my taste.

Comments closed

The Benfit of Disabling V-Order in Fabric Dataflows Gen2

Chris Webb covers a specific use case:

Quite a few new Dataflows Gen2 features were released recently without much fanfare, but that doesn’t mean they aren’t important. I will take a look at them all in my next few posts; in this first post I’ll look at the ability to disable V-Order on staged data.

As the (very detailed) documentation for this new feature describes, V-Order is a write-time optimisation for the parquet files that underpin the Delta tables that OneLake uses to store data. It slows down writing data to the tables but means that reading data from them, for example in Power BI Direct Lake mode, is much faster. 

Click through to see how disabling V-Order can make certain staging loads faster.

Comments closed

Comparative Query Analysis and Planning

Christophe Pettus has a two-parter. The first post covers how a half-dozen data platform technologies handle cost-based analysis:

PostgreSQL has ANALYZE. You run it (or autovacuum runs it for you), it draws a sample of 300 × default_statistics_target rows, and it writes a row per column into pg_statistic: a null fraction, an n-distinct estimate, a most-common-values list, an equi-depth histogram, and a physical-vs-logical correlation. The planner reads those numbers, multiplies selectivities together, costs a handful of join strategies, and picks one. Three join algorithms are on the menu: nested loop, merge join, hash join.

That is the entire shape of the problem, and every cost-based optimizer ever shipped solves the same one. They differ in three places, and only three: where the numbers come from, how stale the numbers are allowed to get, and which plan shapes are even legal to choose between. The algorithms are the boring part. Everybody hash-joins. The interesting part is the bookkeeping.

Then there’s how each of the systems generates a query plan:

Statistics are the input. Planning is what the database does with them: it takes a declarative query, which describes what you want and says nothing about how, and turns it into an executable plan, which is nothing but how. There are two jobs inside that. First, rewrite the query into a logically equivalent but more tractable shape, which is where subquery flattening, predicate pushdown, and view merging live. Second, search the space of physical plans (join orders, join algorithms, access paths) for the cheapest one the cost model can find. The second job is the hard one, because the number of possible join orders for a query grows faster than anyone wants to contemplate, and every database in this article is, underneath, a strategy for not enumerating all of them.

Two questions separate the six systems here. How does each one tame that search space? And once it has an answer, how much will it let you argue with the result? Those sound like the same question. They are not, and the most useful thing this comparison does is pull them apart. A database can search brilliantly and refuse you any override at all (Snowflake), search crudely and hand you a fistful of hints anyway (MySQL until recently), or search hard and expose every lever ever machined (Oracle). Sophistication of the search and generosity of the control surface are independent axes. Knowing where a system sits on each tells you most of what its planner feels like to live with.

Slightly odd is that there’s a section of DB2 but not on SQL Server. But it is a good cross-comparison of several of the top relational database options.

Comments closed

Optional Parameter Plan Optimization

Hugo Kornelis concludes a mini-series:

It’s time to finish the triptych on bad parameter sniffing, and how Microsoft tries (and fails) to fix this for you.

After first talking about bad parameter sniffing in general, I used my last video to explain Parameter Sensitive Plan Optimization, the feature that Microsoft released in SQL Server 2022 as an attempt to fix one of the three root causes for bad parameter sniffing. If you want to see why I consider that a failed attempt, go watch that video.

In SQL Server 2025, Microsoft then added Optional Parameter Plan Optimization (OPPO). Their attempt to fix the second of those three root causes. Put where PSPO in most cases simply doesn’t do anything, or does do something, but not enough, OPPO does too much. It fixes bad parameter sniffing due to optional parameters in demos that are carefully curated to showcase the feature, but it does more bad than good in more realistic scenarios.

Read on for Hugo’s overview, as well as a video.

Comments closed