Press "Enter" to skip to content

Curated SQL Posts

Removing Duplicates with Power Query

Published 2020-02-12 by Kevin Feasel

Matt Allington wants to remove duplicate records while retaining the last version of the record using Power Query:

Today I was helping a customer with a problem that seemed quite simple on the surface.  She had a data table containing historical customer sales orders (each customer has many orders on different dates).  The objective was to filter this table in Power Query and just load one record for each customer – the one that was the last order date.  To illustrate the problem more clearly, I have adapted the scenario using the Adventure Works database.

Click through for the demo, and also read the comments as there are a lot of interesting attempts to solve the problem there as well.

Comments closed

Solving Groups of Groups Problems with Window Functions

Published 2020-02-12 by Kevin Feasel

James McGillivray shows off some of the power of window functions:

Windowing functions are an underused feature in SQL Server, with myriad uses. The most common problems we solve are running totals, seeing group totals on the same line as individual lines from the group (allowing calculations like subtotal %). I don’t know if there are better solutions, but before Windowing Functions, I used to solve these kind of problems with self joins, or nested queries. The performance of Windowing Functions is significantly better than this approach, and that alone has made my life considerably better.

The more I learn about Windowing Functions, the more often I see use cases where they are useful.

I love talking about, and teaching people to use, Windowing functions to make their lives better, so I’m quite excited to be able to use them as the topic for today’s post.

Specifically, James looks at groups of groups problems and aggregates of aggregates problems.

Comments closed

Uploading Multiple RDL Files to SSRS

Published 2020-02-12 by Kevin Feasel

Stuart Ainsworth shows us how to push several SSRS reports at a time:

My QA folks have a problem; development has been working on migrating a bunch of reports from a legacy 3rd-party application to SQL Server Reporting Services. Development has finished their “sprint” (*cough* waterfall), and handed over 52 reports to QA, and told the to load them into their QA server. The problem? The web interface only loads 1 report at a time.

Me, being the DevOps guy that I am, thought “well, that’s just silly; there has to be a way to upload all of the reports at once, rather than clicking on some silly buttons over and over again”.

Read on to see what Stuart did next.

Comments closed

Unit Testing in Apache Flink

Published 2020-02-11 by Kevin Feasel

Kartik Khare has a guide to help us write unit tests for our Apache Flink code:

Writing tests for process functions, that work with time, is quite similar to writing tests for stateful functions because you can also use test harness. However, you need to take care of another aspect, which is providing timestamps for events and controlling the current time of the application. By setting the current (processing or event) time, you can trigger registered timers, which will call the onTimer method of the function

Click through for demos and more details on the test harness.

Comments closed

Finding the Right Disk and Data Node Sizes in HDFS

Published 2020-02-11 by Kevin Feasel

Lokesh Jain has some advice when it comes to disk and data node size:

There are two factors to keep in mind when choosing node capacity. These will be discussed in detail in the next sections.

1. Large Disks – total node capacity being the same, using more disks is better as it yields higher aggregate IO bandwidth.
2. Dense Nodes – as nodes get denser, recovery after node failure takes longer.

These factors are not HDFS-specific and will impact any distributed storage service that replicates data for redundancy and serves live workloads.

Click through for specific advice on maximum disk and node sizes.

Comments closed

Pulling R Packages from Fedora

Published 2020-02-11 by Kevin Feasel

Inaki Ucar has an interesting project:

Bringing R packages to Fedora (in fact, to any distro) is an Herculean task, especially considering the rate at which CRAN grows nowadays. So I am happy to announce the cran2copr project, which is an attempt to maintain binary RPM repos for most of CRAN (~15k packages as of Feb. 2020) in an automated way using Fedora Copr.

Click through for installation instructions if you’re using an RPM-based Linux distribution like Fedora or CentOS. H/T R-Bloggers.

Comments closed

Scripting and Deploying SQL Agent Jobs

Published 2020-02-11 by Kevin Feasel

Alex Yates shows how you can incorporate SQL Agent jobs in your CI/CD process:

Basically, we need to put all the SQL Agent Job .sql scripts into a git repo. Then we need a PowerShell script that executes each .sql script against the necessary target databases. If you use SSDT, you might prefer to use a post deployment script to do this. That bit should be reasonably straight forward. I’ll leave that as a task for the user since I’m short on time.

You probably want to put some thought into whether your agent jobs are scoped to a particular database, general server admin for a specific server, or whether you want them to be standardised across many servers since this may affect where you choose to put your jobs ion source control and on what schedule you want to deploy them.

It may also make sense to set up MSX if you have a central server. That would make Agent job deployment easier and you can still script out which sets of servers get which jobs.

Comments closed

Tenant Usage Monitoring with Power BI

Published 2020-02-11 by Kevin Feasel

Jeff Pries shows us the culmination of several blog posts’ worth of work:

Most of those posts (listed at the bottom of this page) are fairly long and technical — as, initially getting started using a program to read Power BI data via the API can be a bit much.

This post is going to be a bit different. Short and sweet. The payoff for all that hard work authenticating to Power BI, requesting data, downloading that data, and storing it in an easy to use SQL table.

With all of the hard work out of the way, its time to build a Power BI report to explore that great Activity Log usage data.

This is the payoff and it’s quite useful.

Comments closed

Benford’s Law in Power BI

Published 2020-02-11 by Kevin Feasel

Imke Feldmann shows how you can build up a Benford distribution in DAX:

The green columns show how often each number should be the first digit in numbers that should follow the Benford-distribution. In black you’ll see the actual distribution of first digits within my table. Lastly, the red line shows the percentual absolute deviations between actual and Benford values.

In this example, there is a relatively high occurrence of numbers starting with 4 and 5. So this could be a sign for fraudulent manipulations.

In the example, eyeballing it says things look pretty good. It’s interesting to see just how many things fit a Benford distribution, including populations, budgets (when you have enough line items), expenses, etc. Not everything does, however—high and low temperatures tend not to, either in Fahrenheit or Celsius.

Comments closed

The Histogram Output with Extended Events

Published 2020-02-11 by Kevin Feasel

Grant Fritchey talks us through the histogram output in Extended Events:

The histogram target behaves similarly to the event_counter target. The event_counter target counts the number of times that an event occurs. However, the histogram target lets you pick a grouping mechanism for the histogram. You can use either an action, or an event field.

For demonstration purposes, what I want to know is, per object in the database, how many times are the statistics automatically updated?

Read on to see the test, including event setup, data-building queries, and usage of the histogram itself.

Comments closed