Press "Enter" to skip to content

Day: August 10, 2022

Anomaly Detection over Delta Live Tables

Avinash Sooriyarachchi and Sathish Gangichetty show off an interesting scenario:

Anomaly detection poses several challenges. The first is the data science question of what an ‘anomaly’ looks like. Fortunately, machine learning has powerful tools to learn how to distinguish usual from anomalous patterns from data. In the case of anomaly detection, it is impossible to know what all anomalies look like, so it’s impossible to label a data set for training a machine learning model, even if resources for doing so are available. Thus, unsupervised learning has to be used to detect anomalies, where patterns are learned from unlabelled data.

Even with the perfect unsupervised machine learning model for anomaly detection figured out, in many ways, the real problems have only begun. What is the best way to put this model into production such that each observation is ingested, transformed and finally scored with the model, as soon as the data arrives from the source system? That too, in a near real-time manner or at short intervals, e.g. every 5-10 minutes? This involves building a sophisticated extract, load, and transform (ELT) pipeline and integrating it with an unsupervised machine learning model that can correctly identify anomalous records. Also, this end-to-end pipeline has to be production-grade, always running while ensuring data quality from ingestion to model inference, and the underlying infrastructure has to be maintained.

Click through to see their solution using Databricks and delta lake.

Comments closed

Power BI Desktop August 2022 Updates

Matt Allington looks at some recent updates to Power BI:

I’ve been pretty busy over the last few months. The demand for Power BI skills has never been stronger, and my company is super busy. I haven’t written a blog article for a while, but I wanted to take a bit of time out this morning to talk about the August 2022 update to Power BI Desktop. As Power BI matures, there is less and less to get excited about with a new release of Desktop, but there were a couple of things that caught my eye in this release, worthy of calling out.

Read on for a couple of quality of life improvements.

Comments closed

Finding Guidance on Power BI

James Serra puts together a compendium:

Recently there has been a number of great articles published on Power BI that I wanted to make you aware of that go beyond the features descriptions found in the Power BI documentation. These new articles fall under the Power BI guidance documentation and are designed to address common strategic patterns.  Below is my summary of the articles, and check out Power BI guidance from the CAT by Matthew Roach for a more detailed summary.

If you’re interested in Power BI administration and strategic deployment, there’s a lot of good information here.

Comments closed

Formatting Binary LSN Values

Michael J. Swart does a bit of shuffling:

Typically as developers, we don’t care about these values. But when we do want to dig into the transaction log, we can do so with sys.fn_dblog which takes two optional parameters. These parameters are LSN values which limit the results of sys.fn_dblog. But the weird thing is that sys.fn_dblogis a function whose LSN parameters are NVARCHAR(25).

The function sys.fn_dblog doesn’t expect binary(10) values for its LSN parameters, it wants the LSN values as a formatted string, something like: 0x00000029:00001a3c:0002.

Never fear, though: Michael’s got us covered. Click through for a conversion function.

Comments closed