Press "Enter" to skip to content

Month: February 2022

From Cosmos DB to Dedicated SQL Pools via Synapse Link

Jovan Popovic shows off Azure Synapse Link:

At the time of writing this article, the dedicated SQL pool doesn’t have the ability to read data from CosmosDB/Dataverse using the Synapse link. There are scenarios where you would need to use CosmosDB data in your dedicated SQL pool, so you would need to find a way how to load data. In theory, you could create an ADF pipeline that reads data from CosmosDB or Dataverse and store data in the dedicated SQL pool as a target. This might be a problem if your Pipeline is reading data directly from CosmosDB account because it might impact both operational workload performance and cost. The analytical storage is the recommended location that you should use to fetch all data from CosmosDB/Dataverse.

In this post, I will describe how to use a two-step approach where you export your data using the serverless SQL pool via Synapse link into Azure Data Lake storage, and then load data into the dedicated SQL pool table. This process is shown in the following figure:

A couple of weeks back, I wrote about another method of doing this through the Spark pool. Now you have two options.

Comments closed

Restoring PostgreSQL Backups in Azure

Grant Fritchey tests a restore plan:

I recently wrote an article about PostgreSQL restores (and by extension, backups) over on Simple-Talk. The restore process within PostgreSQL, without 3rd party involvement, can be a little tricky. However, when you are using a Platform as a Service offering, like Azure Database for PostgreSQL, things get quite a bit easier. Let’s explore this just a little.

Read the whole thing if you’re thinking about PostgreSQL or Azure Database for PostgreSQL.

Comments closed

Change Data Capture and Availability Groups

Jeff Iannucci raises job awareness:

If you’ve ever had to implement Change Data Capture (CDC) for a database in an Availability Group, then you know that the CDC jobs don’t really consider the Availability Group. The capture and cleanup jobs created are set up as if the database exists only on a single instance.

And that’s a problem, because I would guess quite lot of databases are in Availability Groups. Maybe even some of yours. If you have this issue, I’ve put together a step-by-step solution in this post.

Click through for the Microsoft way and the Jeff way.

Comments closed

Filter on Aggregate Columns in Spark

Landon Robinson shows off the HAVING clause:

Having is similar to filtering (filter()where() or where, in a SQL clause), but the use cases differ slightly. While filtering allows you to apply conditions on your data to limit the result set, Having allows you to apply conditions on aggregate functions on your data to limit your result set.

Both limit your result set – but the difference in how they are applied is the key. In short: where filters are for row-level filteringHaving filters are for aggregate-level filtering. As a result, using a Having statement can also simplify (or outright negate) the need to use some sub-queries.

Click through for examples of HAVING in use.

Comments closed

Kibana Dashboards on Azure Data Explorer

Guy Reginiano has an announcement for us:

Elasticsearch and Kibana users can now easily migrate to Azure Data Explorer (ADX) while keeping Kibana as their visualization tool, alongside the other Azure Data Explorer experiences and the powerful KQL language.
A new version of K2Bridge (Kibana-Kusto free and open connector) now supports dashboards and visualizations, in addition to the Discover tab which was previously supported.

Click through to see how it works. I’m not the world’s biggest fan of Kibana by any stretch of the imagination but it’s nice to have this ability.

Comments closed

Working with Notebooks in Azure ML

I have started a new series:

In the prior series, Low-Code Machine Learning with Azure ML, we saw how to get started with Azure Machine Learning in a fairly pain-free way, especially for developers getting started with machine learning. In this series, I will assume that you already know all of those details and instead, we’re going to go full-code.

There are a few different ways in which we can go full-code with Azure ML. Today, we’re going to look at the easiest of those methods: using Jupyter notebooks within Azure ML Studio.

Read on for the first post in the series.

Comments closed

Data Masking with Powershell

Jeff Hicks wants to stop a shoulder-surfing attack:

If I print this or send it to a file, I might not want the computername to be shown. Or maybe even my name. I want PowerShell to handle this for me. In short, I need to replace strings like ‘Prospero’ and ‘Jeff Hicks’ with some like XXX and ‘Joe Doe’.

PowerShell has a Replace operator. Or I can use regular expressions. Naturally, I wenbt with the latter.

Read on to see what you’d need to do.

Comments closed

Modifying Existing Objects with Powershell

Robert Cain dives a bit further into objects in Powershell:

So far we’ve covered a lot in this series on PowerShell Objects. We began by creating a class using the class keyword introduced in PowerShell 5. We then looked at using PSCustomObject to create new objects, then add methods to those objects. In the previous post, we used C# code to create our PowerShell objects.

In this installment, we’ll see how to add properties and methods to existing objects created by someone else. For this example we’ll use objects returned by the Get-ChildItem cmdlet, but we could use any objects.

Click through to see what you can do.

Comments closed