Press "Enter" to skip to content

Author: Kevin Feasel

Implementing IoT-Style Data in Microsoft Fabric

Published 2026-06-24 by Kevin Feasel

Hristo Hristov takes us through a walkthrough:

Hardware sensors or diverse types of equipment can generate IoT data at a high frequency, e.g., every second. Additionally, IoT data can be messy, semi-structured or just have huge volume and many disparate sources. How to ingest and model IoT data in Microsoft Fabric using the medallion lakehouse architecture?

As I was reading through this, the thing that kept coming to my mind is, if we’re really working with device data at a fairly high periodic frequency (e.g., once a minute or more often), this is probably a job for the Eventhouse and KQL. Though if your devices either don’t collect push information more frequently than, say, hourly, this approach is probably fine.

Comments closed

Thoughts on Deploying Fabric Data Agents

Published 2026-06-24 by Kevin Feasel

Marc Lelijveld performs a deployment:

Over the past year, I’ve frequently blogged about Fabric Data Agents. Alongside myself, many other community members have been sharing their experiences and best practices to get the most out of Data Agents. However, there is one topic I rarely see discussed: deployment of Data Agents.

As Data Agents become part of production-grade solutions, deployment and lifecycle management become increasingly important. Building a Data Agent is one thing, but moving it consistently between Development, Test, and Production environments is a completely different challenge.

In this blog, I will share my current best practices around deploying Fabric Data Agents, including what works today, where the limitations are, and the gaps that still exist.

Read on to see what Marc recommends at this time, with the proviso that some of this will likely change as the product develops further.

Comments closed

Spark DataFrameWriters

Published 2026-06-23 by Kevin Feasel

Miles Cole compares two generations of DataFrameWriter:

Most Spark developers learn to write data with df.write long before they ever encounter df.writeTo. It is simple, familiar, and everywhere: choose a format, pick a mode, add a few options, and save the result to a table or path. For years, that mental model worked well enough. Spark was often writing files first and tables second.

But modern lakehouse systems have changed the contract.

Read on to learn how, and what common problem the DataFrameWriterV2 is there to solve.

Comments closed

The Importance of Testing Received Wisdom

Published 2026-06-23 by Kevin Feasel

Mark Wilkinson lays out an argument:

Life is full of “absolutes”. For example, the Star Trek: The Next Generation episode “The Measure of a Man” is often cited as the best episode of the series, and many folks will tell you that you should never adjust max worker threads. But once you take the time to dig in, you realize that “Darmok” is in FACT the best episode of ST:TNG, and you’ll also find a small cohort of folks adjusting max worker threads on all of their SQL Server instances. Are these people just abnoxious contrarians? No. They just did their own testing to validate the common wisdom.

Click through for an example from Mark around 64K allocation unit sizes for NTFS volumes. And I’ll give one on max worker threads. I had a consulting client at one point which had per-customer databases. Each customer was, in general, quite small, so they had thousands of databases on the instance. They also wanted high availability on the system, so they wanted each database mirrored to a different server.

If they didn’t spike max worker threads to extreme levels, the server would have fallen over simply from the weight of all of the open database mirroring connections. The actual server workload was fine and it could handle all of the open worker threads because the large majority were doing nothing. But if a zealous problem-solver popped in, ran a diagnostic, saw that they were violating “best practices,” and “fixed” the problem, that would have been a bad day.

Unrelated but similar story: the one time they did need to fail over due to an emergency, it was also a bad day. Because even if the instances can handle 2500+ databases, it turns out that having them all fail over at the same time on low-powered Azure hardware was not a pleasant experience.

Comments closed

Optimizing Polymorphic Associations in Postgres

Published 2026-06-23 by Kevin Feasel

Andrei Lepikhov continues a thread:

Recently, I looked into how common polymorphic associations actually are in relational databases — a performance-hostile pattern built around a discriminated foreign key that ORMs (Rails, Django, Hibernate), CRM platforms (Salesforce), and 1C generate automatically. The front page of a typical online store, or the activity feed of a CRM, is built by exactly this kind of query: a base table is LEFT JOIN-ed to every possible subtype through a (type, id) pair of columns.

That earlier article answered the question ‘how widespread is this pattern?’ After all, if you’re going to improve something, it helps to know how useful the improvement will be, right? Here, I want to give a sense of how this pattern leads to performance regressions and point out directions in the PostgreSQL optimiser that could make the situation easier.

Much of this is speculative in nature but the three proposed solution ideas are all interesting.

Comments closed

Patched SQL Injection Vulnerability in sys.sp_dbmmonitorupdate

Published 2026-06-23 by Kevin Feasel

Fabiano Amorim digs into a fixed issue:

What makes this case particularly interesting is not just that the vulnerability exists in a trusted system object, but how it works: the injection bypasses a REPLACE-based sanitization attempt through a subtle Unicode character conversion that happens silently during a variable assignment.

The vulnerability was reported to Microsoft and they have since fixed it, but it’s still worth exposing and explaining given how intricate it is. So, that’s what I’ll do in this article.

Click through to see how it works. And of course this database mirroring stored procedure is still hanging around long after database mirroring itself was deprecated. But that’s the downside to deprecation without subsequent removal.

Comments closed

The State of SQL Notebooks

Published 2026-06-22 by Kevin Feasel

Deborah Melkin takes a look at the state of notebooks in the Microsoft ecosystem:

Azure Data Studio, as some of you know, has now been deprecated. That came out a while ago and the official deprecation was, I think, a couple months ago at this point. It’s all a blur, but needless to say, no more Azure Data Studio. I had gotten an email from someone who said they saw my presentation and they’d love to see more about it, especially with VS Code. Because there was an extension in VS Code for notebooks, and particularly something called .NET Interactive, which are polyglot notebooks, polyglot multi-language. All right, that was really cool and I had started addressing that, too, because it had just been introduced. It’s a really cool concept.

And then before I had a chance to put it together, new notice from Microsoft. Guess what’s being deprecated?

You guessed it. .NET Interactive notebooks. They went bye-bye. Great.

But wait, I hear SQL Notebook’s theme music?

Comments closed

User-Defined Functions and Power BI Testing

Published 2026-06-22 by Kevin Feasel

John Kerski is excited:

User Defined Functions (UDFs) are, in my opinion, the biggest update to Power BI Desktop since PBIP.

That may sound dramatic, but if you care about DataOps, semantic model quality, and reusable development patterns, UDFs fundamentally change what is possible with DAX.

Reuse is one of the core principles of DataOps. For years we have been able to build reusable patterns in Power Query, PowerShell, Python, YAML, and infrastructure automation. But DAX was always missing a key capability: reusable logic that could live inside the semantic model itself.

Until now.

Read on to learn more, as well as to get a link to John’s PQL.Assert DAX unit testing library.

Comments closed