Press "Enter" to skip to content

Month: November 2023

Last Observation Carried Forward in SQL Server 2022

Published 2023-11-16 by Kevin Feasel

Barney Lawrence shows off a nice enhancement to T-SQL in SQL Server 2022:

With SQL Server 2022 came a much requested additional feature added from the SQL standards – IGNORE_NULLS. You can probably guess what it does. Drop in IGNORE_NULLS after your function and you can blur the non null values over those gaps giving us results like this:

Read on for the pre-2022 version of the query and what it does, versus the version with IGNORE_NULLS specified. This small flag is extremely helpful in time series statistical analysis and I’m glad it’s in SQL Server now.

Comments closed

Microsoft Fabric Now GA

Published 2023-11-16 by Kevin Feasel

Ben Jarvis announces the news:

On the 15th November Microsoft announced the general availability of Microsoft Fabric. Having seen Microsoft Fabric develop through our involvement in the private and public preview this is a really exciting announcement and all of us at Adatis are looking forward to seeing how it will fit into the solutions we are deploying to customers. 

It makes sense to announce going GA at Ignite, though I don’t think Fabric is ready for prime time yet. There are still so many gaps in the product, and even though many items are on the roadmap, I wonder if Fabric will get a bad reputation from going GA too early. My analogy is Cyberpunk 2077: when the game came out, it was a buggy mess. Most of the bugs have been fixed now, a couple of years later, but for many people, their everlasting impression of the game is how buggy it was.

Comments closed

Execution Plans for Window Functions

Published 2023-11-16 by Kevin Feasel

Hugo Kornelis digs in:

In his invitation, Steve specifically asks for examples where Window functions provided a neat solution to a real world problem. Well, sorry Steve, but I am not going to do that. But your invitation did inspire to me write about the execution plans for these window functions. And there is, in fact, so much to write about it, that this is just the first part.

So that makes this post not only a T-SQL Tuesday contribution, but also part 23 in my ongoing plansplaining series: blog posts where I take an in depth look at execution plans to explain how exactly they work, and point out often overlooked details. In this post, I will look at the basics of window functions, as they have existed for over 10 years now. I will point out a few interesting performance gotchas. And there are some links to feedback items that you can upvote (if you are so inclined) to pressure Microsoft to make some changes to the optimizer and the execution engine, to get some small but still welcome performance gains.

Read on to get an in-depth understanding of how those window functions actually operate once the database engine gets ahold of your query.

Comments closed

Updates to Azure Well-Architected Review Assessments

Published 2023-11-15 by Kevin Feasel

Stephen Sumner shows off some changes:

Microsoft is excited to announce a significant update to the Azure Well-Architected Review assessment helps you build and optimize workloads. It walks you through a series of questions about your workload. Based on your responses, it generates tailored and prioritized recommendations to improve your workload design. The guidance is actionable and applicable to nearly every workload. It aligns with the latest best practices across the five key pillars of reliability, security, cost optimization, operational excellence, and performance efficiency (see figure 1).

I’m a big fan of the Well-Architected Framework and the assessments Microsoft has put together. An assessment can take teams within a company days to complete because the questions are so thorough, but once you do get through the list, you’ll get some great practical insights on your setup and what you can do to improve performance and save money.

Comments closed

Using the WINDOW Operator in SQL Server 2022

Published 2023-11-15 by Kevin Feasel

Steve Jones cuts out the duplication:

I am the host for T-SQL Tuesday this month, and I hope that a lot of people like the topic. This idea actually came to me earlier this year when I happened to see someone ask about a T-SQL problem and get an answer using a Window function. This person mentioned they hadn’t used the window function before, and I wondered how many people haven’t even tried using the OVER() clause with a window function.

Read on to see how you can use WINDOW to replicate window frames, which can be quite helpful for larger queries. It doesn’t change the performance profile but does make the code more readable.

Comments closed

Operations Auditing with LAG() and LEAD()

Published 2023-11-15 by Kevin Feasel

Deborah Melkin builds a report query:

I’ve been doing a lot of work recently where I have long running processes where I need to have visibility on where we are during the process as well as being able to provide a report we can use to find out if there were any errors, how long each step took, and other useful information that we can use later. My solution was to create an audit log table where I simply added a row before and after each step, recording the information I had available. When the processes are done, I create a query to run over the table returning the values I want to see.

Deb also throws in the only ordered set function in SQL Server, STRING_AGG().

Comments closed

The Power of LAG and LEAD

Published 2023-11-15 by Kevin Feasel

Rod Edwards shows off a great use case for LAG():

I often find myself using windows functions in order to group data in wierd and wonderful ways that a simple GROUP BY can’t do… however, the example below is one I came across quite recently on my travels, I was asked to have a look at some code for optimization opportunities. And in this case, thankfully there were some.

Read on for a lengthy cursor, followed by a much less lengthy (and much faster) window function.

Comments closed

Good Use Cases for Window Functions

Published 2023-11-15 by Kevin Feasel

Aaron Bertrand build a list:

When I first used window functions back in SQL Server 2005, I was in awe. I had always used inefficient self-joins to calculate things like running totals, and these really didn’t scale well with size-of-data. I quickly realized you could also use them for ranks and moving averages without those cumbersome self-joins, elaborate sub-queries, or #temp tables. Those all have their place, but window functions can make them feel old-school and dirty.

I’d also recommend learning more about the APPLY operator as well, as it can, depending on the circumstances, be even more effective than window functions (combined with common table expressions) for some of the use cases.

Comments closed

Creating Prediction Intervals in R

Published 2023-11-14 by Kevin Feasel

Steven Sanderson builds a prediction interval:

Prediction intervals are a powerful tool for understanding the uncertainty of your predictions. They allow you to specify a range of values within which you are confident that the true value will fall. This can be useful for many tasks, such as setting realistic goals, making informed decisions, and communicating your findings to others.

In this blog post, we will show you how to create a prediction interval in R using the mtcars dataset. The mtcars dataset is a built-in dataset in R that contains information about fuel economy, weight, displacement, and other characteristics of 32 cars.

Click through to see an example based on linear regression.

Comments closed

Searching for Tenant Settings in Microsoft Fabric

Published 2023-11-14 by Kevin Feasel

Wolfgang Strasser does a search:

Another nice feature update for administrators hit Microsoft Fabric – tenant settings got a search box!

Before the this new feature was added to the tenant settings page in October, you had one chance to search for the right tenant setting in the long list – the browser search feature. Just hit CTRL + F and search for i.e. Excel.

This is a quick quality of life improvement.

Comments closed