The data lake introduces a new data analysis paradigm shift:
OLD WAY: Structure -> Ingest -> Analyze
NEW WAY: Ingest -> Analyze -> Structure
This allows you to avoid a lot of up-front work before you are able to analyze data. With the old way, you have to know the questions to ask. The new way supports situations when you don’t know the questions to ask.
This solves the two biggest reasons why many EDW projects fail:
Too much time spent modeling when you don’t know all of the questions your data needs to answer
Wasted time spent on ETL where the net effect is a star schema that doesn’t actually show value
There are some good details here. My addition would be to reiterate the importance of a good data governance policy.
Big data comes in a variety of shapes. The Extract-Transform-Load (ETL) workflows are more or less stripe-shaped (left panel in the figure above) and produce an output of a similar size to the input. Reporting workflows are funnel-shaped (middle panel in the figure above) and progressively reduce the data size by filtering and aggregating.
However, a wide class of problems in analytics, relevance, and graph processing have a rather curious shape of widening in the middle before slimming down (right panel in the figure above). It gets worse before it gets better.
In this article, we take a deeper dive into this exploding middle shape: understanding why it happens, why it’s a problem, and what can we do about it. We share our experiences of real-life workflows from a spectrum of fields, including Analytics (A/B experimentation), Relevance (user-item feature scoring), and Graph (second degree network/friends-of-friends).
The examples relate directly to Hadoop, but are applicable in other data platforms as well.
Once saved, the Power BI file size was 289MB! Is this good for 10 million rows? It’s certainly better than the 360MB CSV file but not by much. Certainly not close to the 10:1 compression claimed to be achievable using the SSAS Tabular engine used by Power BI.
I think we can do better than that….
Read on to see the specific optimizations, turning this from a 289 MB data model into a 9 MB data model.
Although traditional dimension modeling – as explained by Ralph Kimball – tries to avoid snowflaking, it might help the processing of larger dimensions. For example, suppose you have a large customer dimension with over 10 million members. One attribute is the customer country. Realistically, there should only be a bit over 200 countries, maximum. When SSAS processes the dimension, it sends SELECT DISTINCT commands to SQL Server. Such a query on top of a large dimension might take some time. However, if you would snowflake (aka normalize) the country attribute into another dimension, the SELECT DISTINCT will run much faster. Here, you need to trade-off performance against the simplicity of your design.
There are several good tips here.
Working with role playing dimensions, which are found when you have say multiple dates in a table and you want to relate them back to a single date table, have always been problematic in SQL Server Analysis Services Tabular. Tabular models only allow one active relationship to a single column at a time. The picture on the left shows how tabular models represent a role playing dimension, and the model on the right is the recommended method for how to model the relationships in Analysis Services Tabular as then users can filter the data on a number of different date tables.
The big downside to this is one has to import the date table into the model multiple times, meaning the same data is imported again and again. At least that was the case until SQL Server 2016 was released. This weeks TSQL topic Fixing Old Problems with Shiny New Toys is really good reason to describe a better way of handling this problem.
Read on for how to implement calculated dimensions.
It’s clear that in this specific case – with an address column of
nvarchar(60)and a max length of 26 characters – breaking up each address into fragments can bring some relief to otherwise expensive “leading wildcard” searches. The better payoff seems to happen when the search pattern is larger and, as a result, more unique. I’ve also demonstrated why EXISTS is better in scenarios where multiple matches are possible – with a JOIN, you will get redundant output unless you add some “greatest n per group” logic.
Now the complex. To validate that our tree is properly structured, the following statements need to be true:
Each node’s Right value is greater than its Left.
More to the point, each node’s Right value is greater than all of its ancestors’ Left values.
Similarly, each node’s Left value is less than all of its descendants’ Left values (and Right values, obviously!)
Leaf nodes have no gaps between Left & Right:
Right = Left + 1
Depthis easy to verify because we already wrote the rCTE to calculate it!
And of course, no orphans – all ParentIDs lead to an actual parent node, except of course if they’re
Read on for further explanation of these points.
Depth is pretty simple to add if you’ve already got a tree full of data. We can use a recursive common table expression, or “rCTE“. While normally these are frown-worthy (remember, recursion is not SQL’s strong suite), we’re only using it one time to populate an existing data-set, so we can keep on smiling.
Hierarchies in SQL are an important but not well understood topic.
The initial proposed solutions to construct the subgraphs were essentially procedural traversal, dumping pairs of nodes into a temp table and incrementing a counter.
Let us try getting out of a procedural mindset and starting to think in sets instead. Let us give each subgraph a name, and a member node. Essentially, this directly models. The diagram I just gave you a paragraph or two ago. The next question is how do we get names for the subgraphs. I will propose the simple solution that each subgraph takes the name of the lowest element in it. This would give us a table that looks like this:
Read the whole thing. Given the recent revival of graph databases, it’s important to take note that you can model the network-style problems using either graphs or relations; the trick is figuring out which is going to give you better long-run performance.
I recently had the need to compare a “Gold” or “Master” copy of our database to the 300 client copies of that database that exist in our Production environment. I’m not alone…many fellow DBAs have had the same need. Google searches for this confirm it. This is for an upcoming upgrade to the application that will need post-upgrade comparison/verification.
There are 3rd party tools that do SQL Compares…my particular favorite is aptly named SQL Compare from Red Gate. I’ve been using it off an on for 10 years. I don’t know if it can be set up to hit more than one database at a time. The other issue is that I don’t have a copy here.
Microsoft’s SQL Server Data Tools will also do this within Visual Studio. Still one database at a time. I forget where, but someone pointed me to the fact that SSDT uses SQLPackage.exe under the hood to do the work. I figure if I can run it at a command line I can script out all of the databases. I’m not much of a DOS scripting guy, so everything that follows is just my hack version…but it works, and not just on my machine!
This is very useful if you work in an environment with multiple copies of databases—or even if you have dev, test, and prod versions of the same database.