A Fine Slice Of SQL Server
I know that some of this might be able to be done with other calculation groups. I find I have more flexibility when combining Calculation Groups with a disconnected table.
Below are some of my previous calculation group blog posts that might also be of interest:
Click through for the demo.
Comments closedSander Stad walks us through some changes to the data masking algorithm in dbatools:
If you’ve used the data masking command in dbatools you’ve probably noticed that the PowerShell session becomes memory intensive when it has to handle larger tables with one or more unique indexes.
The reason that happens is that during the data masking process the command looks for any unique indexes in the table. If it finds a unique index it will create a unique row for all the columns in the unique index.
Read on to see how Sander handled this.
Comments closedGarland MacNeill takes us through a replication issue:
For the past couple of days, I’ve been working on getting transactional replication set up between a couple of servers in between other projects I’ve been working on. For the last day I kept running into the following error:
“The remote server <“server name”> does not exist, or has not been designated as a valid Publisher, or you may not have permissions to see available Publishers. ”
Click through for the solution.
Comments closedOliver Koernig walks us through some of the basics of MLOps using MLflow and Azure Databricks:
Most organizations today have a defined process to promote code (e.g. Java or Python) from development to QA/Test and production. Many are using Continuous Integration and/or Continuous Delivery (CI/CD) processes and oftentimes are using tools such as Azure DevOps or Jenkins to help with that process. Databricks has provided many resources to detail how the Databricks Unified Analytics Platform can be integrated with these tools (see Azure DevOps Integration, Jenkins Integration). In addition, there is a Databricks Labs project – CI/CD Templates – as well as a related blog post that provides automated templates for GitHub Actions and Azure DevOps, which makes the integration much easier and faster.
When it comes to machine learning, though, most organizations do not have the same kind of disciplined process in place.
Read on for a demonstration of the process.
Comments closedHolger von Jouanne-Diedrich takes us through Dijkstra’s algorithm for shortest path calculations:
This post is partly based on this essay Python Patterns – Implementing Graphs, the example is from the German book “Das Geheimnis des kürzesten Weges” (“The secret of the shortest path”) by my colleague Professor Gritzmann and Dr. Brandenberg. For finding the most elegant way to convert data frames into igraph-objects I got help (once again!) from the wonderful R community over at StackOverflow.
Dijkstra’s algorithm is a recursive algorithm. If you are not familiar with recursion you might want to read my post To understand Recursion you have to understand Recursion… first.
Click through for an implementation in R.
Comments closedJack Vamvas shows how you can check on the SQL Server Agent status using T-SQL:
I want to check if SQL Server Agent is running using t-sql. I know I can check through the services.msc or other methods such as Powershell scripts to report on the SQL Server Agent status , but the requirement is to extract this information via t-sql. Do you have a script using t-sql to get the SQL Server Agent Status?
There is a method and Jack shows us the way.
Comments closedMohammad Darab shows off a tool to monitor the Kubernetes cluster driving a Big Data Cluster:
I don’t recall how I came across this Kubernetes IDE called Lens, but all I know is it’s cool as hec! It connects to a Kubernetes cluster (using the kube config file) and gives you an in depth view of all the different Kubernetes objects, their associated yaml files, health/metrics, etc. In this blog post I will show you how we can look into a Big Data Cluster’s Kubernetes infrastructure using Lens.
Click through for instructions on installation, as well as how to use the product.
Comments closedMartin Schoombee gets off the beaten path:
We’ve only used the PowerShell cmdlets for Power BI so far in this series, but things are about to get interesting because there aren’t cmdlets available for everything you might want to do. One such thing is updating parameters, and we’re going to use the Power BI REST API (which the cmdlets use underneath the covers anyways) to achieve that.
REST APIs are usually a little tricky to deal with, especially the process of authentication. Fortunately there is an Invoke-PowerBIRestMethod cmdlet that makes it possible to use the API in PowerShell without the need to deal with some of the underlying complexities.
Read on to see how the whole process works.
Comments closedSo what’s the hard thing I want to help make easier in this post? Metadata driven pipelines in Azure Data Factory! I had the opportunity awhile back to work with a customer who was pulling data out of large SQL Servers to eventually land data into Azure Synapse SQL pools back when they were still Azure SQL DW. We created a couple load pattern pipelines that used metadata in Azure SQL DB to load Synapse sql pool tables from parquet files in Azure Data Lake Storage (ADLS) Gen 2.
Not gonna lie, the pipelines weren’t easy for me to learn to setup initially. Big thanks to Catherine for your blog which was a life preserver in the hardest parts! So I wanted to see if I could automate it in my old friend PowerShell.
It would also be worth looking at some of the work Paul Andrew has done around ADF.procfwk for another approach to the problem.
Comments closedIf you’ve used Power Query long enough, you’ve needed to group data and provide some sort of summary. For example, grouping by customer and summarizing sales. The normal way the Table.Group function works is to group the entire table by the field(s) you are grouping by, then providing whatever aggregations you need.
However, you can alter that behavior and group data by a column, and have it provide a separate group at each change in the data.
Click through for the rest of the story. There’s some complexity to the problem which is hard to summarize.
Comments closed