Press "Enter" to skip to content

Category: T-SQL Tuesday

Skills for Cloud-to-On-Prem Migration

Reitse Eskens focuses on a set of skills:

This month, Koen Verbeeck invites the blogging community to write about their thoughts on returning to on-premises. What could be struggles, things we have to re-learn, etcetera.

When I read the invite, it immediately sparked inspiration, because there are increasing rumours around cloud exits. People musing about ‘what if’. Some clients reference these questions, but so far no one has directly asked me one with the intent of moving forward with it.

Click through for Reitse’s thoughts.

Comments closed

The Importance of Working with People

Mark Wilkinson hits migration from a different angle:

I haven’t written a blog post (here) in over 5 years, so what better time to start things back up than a T-SQL Tuesday? Big shout out to Koen Verbeeck for hosting this month, and picking a great topic: Back to on-prem?

As someone that just ended a 10+ year stint managing a hybrid environment, this topic is very near and dear to me. I went back and fourth on what to write about for this one. There are a lot of great topics. Reliability and observability almost won out but instead I landed on maybe an unexpected topic: soft skills.

Regardless of whether your company is fully on-prem, fully in the cloud, fully hybrid, or fully without a clue, Mark’s advice hits home. And is also one of those things I’ve struggled with.

Comments closed

T-SQL Tuesday 198 Round-Up

Meagan Longoria wraps up another T-SQL Tuesday:

Thank you to everyone who participated in T-SQL Tuesday #198! When I wrote the invitation post, I intentionally kept the prompt broad because change detection looks different depending on your source system, your infrastructure, your data volumes, and what you need to do with the changes once you have them. The responses covered SQL Server internals, Microsoft Fabric and Synapse, hashing strategies, metadata-driven frameworks, and Synapse workspace diffing with Python. Here’s a summary of each contribution.

Read on for links to eight responses.

Comments closed

Metadata-Driven Frameworks for Change Detection in Microsoft Fabric

Kevin Chant builds a table:

I had various options for this months contribution due to my experience with various change detection solutions. Including Azure Synapse Link for SQL Server 2022. Which I covered in previous posts. Including one that covered some excessive file tests for Azure Synapse Link for SQL Server 2022.

In the end I decided to cover developing metadata-driven frameworks for Microsoft Fabric. Due to the fact that it is such a hot topic for multiple reasons. One of which is the growing availability of open-source, metadata-driven frameworks for Microsoft Fabric.

Read on for three such frameworks and some advice on how to use them.

Comments closed

Tracking Data Changes

Louis Davidson shows a technique:

A few months ago, I wrote a post about comparing sets of data in SQL, This focussed on a type of challenge that is often a one off challenge. The techniques lists (along with a tool like Redgate’s SQL Data Compare aren’t the point of this post, but they are related because when you do change detection from a source, it is super important to check your results occasionally. Having a copy of the complete source to compare to your destination (even if it is just checksums of the data,) is important.

The techniques that Meagan is asking about this month are more ETL related, where you check a stream of data and sync them as changes are made. Typically, you don’t want to compare all the rows in a set, but just the ones that have changed. At some frequency, give me the changes to one set of data and keep it up to date with another.

I’m still partial to using HASHBYTES() on the fields I care about because I don’t trust modified dates unless I know the only way to access that table is via stored procedure, and all of the stored procedures handle updating the modified date correctly.

For row comparisons, I’d also look at EXCEPT for overall row comparison. You do need to do it in both directions, so the pattern is more like A EXCEPT B UNION ALL B EXCEPT A but it works great and natively handles any NULLs along the way.

Comments closed

Tracking Record Changes in SQL

Andy Brownsword builds a hash key:

The issue: there was no indicator of which records had been modified and as a result the process took way too long, and downstream reporting wasn’t available on time.

After reviewing and stepping through the process it became clear that the vast majority of data didn’t change. This was a daily process handling 12 months of data, yet over 99% had no changes at all. However the process ingested the whole dataset (~250m records) and processed it in SQL.

Click through for an architectural-level discussion. In practice, HASHBYTES() works really well, especially when you use CONCAT() or CONCAT_WS() to put together the columns you care about

Comments closed

T-SQL Tuesday 197 Round-Up

Steve Hughes learns something new:

Thanks to everyone who joined the blog party this month. I noticed three themes in the responses. Every response had one or more of these themes woven into their response.

  1. I learned something.
  2. I discovered ways to improve my presentations.
  3. I get more value in the hallway conversations.

Click through for this month’s participants and what they’ve learned.

Comments closed

Presenting for Impact

Rob Farley tells a story:

I like this topic from the legendary Steve Hughes. It’s been a long time since I’ve seen him, but he was always a thoroughly good guy. We both spoke at conferences back in the heyday of the SQL community, and although his journey has been tougher than most in recent years, he is still impacting the world in amazing ways.

Steve is hosting this month’s T-SQL Tuesday, and asks about what we’ve learned from conference sessions, things which impacted us and how we work. It’s an interesting topic for two reasons – firstly, I enjoy giving conference presentations, and secondly, they’re really not my preferred way of learning.

Rob goes on to talk about conference sessions that caught his interest. One book that helped me considerably in my ability to present is Peter Cohan’s Great Demo! This is, admittedly, for sales presentations rather than technical presentations. However, I think it’s pretty straightforward to map most of the concepts to technical demos, and the advice in the book is great for getting your point across early and letting people make sure they are in the right room at the right time straight from the get-go.

Comments closed

T-SQL Tuesday 195 Round-Up

Pat Wright summarizes this month’s T-SQL Tuesday:

What a great collection of stories all about aging!  I really appreciated everyone take time to reply to the question.  

I’m going to do this a little differently for my summary. I will list the blogs below and provide key summary points for each.  I also created a video summary. I intend to do more of this in the future. 

Click through for the two options.

Comments closed

A Story of Code that Aged Well

Hugo Kornelis wrote some code:

The application had clearly been developed by someone who knew just one tool: SQL Server. So he made the database do all kinds of things that really should have been in the application layer. Call external APIs to monitor the primary process of the company. Send fully formatted emails with invoice information. Send formatted HTML and interpret the callback response as a picked menu item. Yes, all that was handled in SQL Server, the frontend app did nothing more than sending query results to the web server and sending HTTP replies back to the database as a parameter into a stored procedure that handled everything.

Read on to see what Hugo was able to do in a one-year timeframe and what it was like coming back to that same code years later.

Comments closed