A Fine Slice Of SQL Server
I’ve been working with CDC and CT way too much, and even I’m annoyed with how much it’s coming out in blog posts.
I’m going to cover a lot of ground quickly here. If you get lost, or there’s something you don’t understand, your best bet is to reference the documentation to get caught up.
Check it out.
Comments closedSoheil Bakhshi has a great step-by-step walkthrough for setting up Azure Purview:
Microsoft newly announced a piece of very exciting news that Azure Purview now supports Power BI. This is massive news from a data governance point of view. Azure Purview is the next generation of Azure Data Catalog with more metadata discovery power and the ability to use sensitivity labels. After reading the news, I immediately decided to set up my test environment and give it a go. I followed the steps mentioned in this article on the Microsoft documentation website but I faced some difficulties to get it to work. And here we are, another blog post to help you to set up the Azure Purview for Power BI.
Click through for a detailed walkthrough.
Comments closedWolfgang Strasser continues a series on Azure Purview. First up is a look at external connections:
During my tests with Azure Purview I found out, that in my demo accounts (I created multiple to test it) the Management menu was missing some items.
Read on to learn why and how you can rectify this. Then, check out Wolfgang’s take on the Starter Kit:
Very often, when I want to test some new services I miss some infrastructure and environments I can start and play with. I am not talking about creating a new Azure Purview account (see my previous blog post – Creating an Azure Purview account) – I am talking about the data infrastructure to analyze, catalog and gain knowledge out of it.
We could start to create such an infrastructure, BUT: the Purview team create a Starter Kit to quickly create a data estate and configure everything that you can start with Purview within a view minutes.
Read on for one issue (of the self-inflicted variety) Wolfgang ran into during deployment. But it does look like a great way to get started with Purview and build up a relevant demo environment.
Comments closedRobert Blackburn shows us one way to handle bad dates when moving to Spark 3:
Moving from a Spark 2 to a Spark 3 runtime has a lot of benefits including big performance improvements through adaptive query execution, dynamic partition pruning, and other optimizations. Some updates may require you to refactor your code. One of them is Delta tables now use the Proleptic Gregorian Calendar. Isn’t a calendar a calendar? Unfortunately, no. The Julian calendar has discrepancies with old dates. Specifically dates before 1582 and timestamps before 1900. Here we will dynamically update these dates for incoming source files.
If you would like to follow along in detail, I have a sample notebook that uses the community edition of Databricks. The DBC Archive file is here and the source file is here.
Fortunately, this change is unlikely to affect most of us, with perhaps the most common issue being that you used 0001-01-01 as a default date.
Comments closedWolfgang Strasser wants to try out Azure Purview:
Basics – Resource group, purview account name (this cannot be changed afterwards) and the location.
As of today (2020-12-06), there are only 5 Azure regions you can choose from to store the Purview metadata. But – in-region scanning from 16 other Azure regions is available in the preview (source)
This is part one of a multi-part series, so stay tuned for more.
Comments closedWolfgang Strasser gives us a once-over on a new service:
Today, at the Azure Data and Analytics event, a new Azure data governance service called Azure Purview (https://aka.ms/AzurePurview) was presented and made available in a public preview.
I have not had a chance to try the actual service, but I found a very interesting video (Microsoft mechanics video) where I took the following screenshots from.
Read on for Wolfgang’s thoughts. It’s definitely a step up from Azure Data Catalog.
Comments closedDavid Eldersveld takes a look at a new sample data set in Power BI Desktop:
With the October 2020 release of Power BI Desktop, Microsoft has incorporated a sample dataset directly into the product. While there are a few commonly used training and demo datasets already, there are benefits to having a basic starter Financials dataset included.
This data can accelerate the learning experience for new users, particularly those who may only need to focus on authoring reports on existing datasets at their organization and don’t need to immediately know much about data modeling.
Click through for David’s mixed review.
Comments closedPaul Andrew compares two techniques for retrieving metadata:
For file types that don’t contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I’ve used the Data Factory Metadata Activity to do this with its structure option. However, while playing around with Azure Synapse Analytics, specifically creating Notebooks in C# to run against the Apache Spark compute pools I’ve discovered in most case the Data Frame infer schema option basically does a better job here.
Now, I’m sure some Spark people will probably read the above and think, well der, obviously Paul! Spark is better than Data Factory. And sure, I accept for this specific situation it certainly is. I’m simply calling that out as it might not be obvious to everyone
Read on for a comparison of the two techniques.
Comments closedSander Stad walks us through some changes to the data masking algorithm in dbatools:
If you’ve used the data masking command in dbatools you’ve probably noticed that the PowerShell session becomes memory intensive when it has to handle larger tables with one or more unique indexes.
The reason that happens is that during the data masking process the command looks for any unique indexes in the table. If it finds a unique index it will create a unique row for all the columns in the unique index.
Read on to see how Sander handled this.
Comments closedPaul Turley continues a series on doing Power BI the right way:
When designing a new data model, this is typically the first thing I do… For every fact table and for each large dimension table, I create a measure that returns the record count for that table. Users normally think about their data in business terms (like sums, ratios and averages) and not about how many records there are in a table. Record counts are a convenient sanity check for record completeness and distribution across groups; and may also be a good indicator for model size and performance.
Paul takes several passes at the problem, getting a bit deeper into it each time.
Comments closed