Press "Enter" to skip to content

Category: Cloud

From Cosmos DB to Dedicated SQL Pools via Synapse Link

Published 2022-02-04 by Kevin Feasel

Jovan Popovic shows off Azure Synapse Link:

At the time of writing this article, the dedicated SQL pool doesn’t have the ability to read data from CosmosDB/Dataverse using the Synapse link. There are scenarios where you would need to use CosmosDB data in your dedicated SQL pool, so you would need to find a way how to load data. In theory, you could create an ADF pipeline that reads data from CosmosDB or Dataverse and store data in the dedicated SQL pool as a target. This might be a problem if your Pipeline is reading data directly from CosmosDB account because it might impact both operational workload performance and cost. The analytical storage is the recommended location that you should use to fetch all data from CosmosDB/Dataverse.

In this post, I will describe how to use a two-step approach where you export your data using the serverless SQL pool via Synapse link into Azure Data Lake storage, and then load data into the dedicated SQL pool table. This process is shown in the following figure:

A couple of weeks back, I wrote about another method of doing this through the Spark pool. Now you have two options.

Comments closed

Kibana Dashboards on Azure Data Explorer

Published 2022-02-03 by Kevin Feasel

Guy Reginiano has an announcement for us:

Elasticsearch and Kibana users can now easily migrate to Azure Data Explorer (ADX) while keeping Kibana as their visualization tool, alongside the other Azure Data Explorer experiences and the powerful KQL language.
A new version of K2Bridge (Kibana-Kusto free and open connector) now supports dashboards and visualizations, in addition to the Discover tab which was previously supported.

Click through to see how it works. I’m not the world’s biggest fan of Kibana by any stretch of the imagination but it’s nice to have this ability.

Comments closed

Working with Notebooks in Azure ML

Published 2022-02-03 by Kevin Feasel

I have started a new series:

In the prior series, Low-Code Machine Learning with Azure ML, we saw how to get started with Azure Machine Learning in a fairly pain-free way, especially for developers getting started with machine learning. In this series, I will assume that you already know all of those details and instead, we’re going to go full-code.

There are a few different ways in which we can go full-code with Azure ML. Today, we’re going to look at the easiest of those methods: using Jupyter notebooks within Azure ML Studio.

Read on for the first post in the series.

Comments closed

Data Mesh in Azure: Self-Service Infrastructure

Published 2022-02-02 by Kevin Feasel

Paul Andrew continues a series on applying data mesh principles in Azure:

This principal is very broad, so I want to break down the theory vs practice as before. The idea of self-service is always a goal in any data platform and the normal thing for analytics is to focus on this within the context of our data consumption. Whereby a semantic layer technology can be used in a friendly business orientated, drag-drop type environment to create dashboards or whatever.

However, my interpretation of ‘self-serve’ for a data mesh architecture goes further than just the dashboard creation use case. This should not just apply at the data consumption layer, but all layers within the solution and for clarify, not just related to the data itself. Hence the term in this principal ‘data infrastructure as a platform’. This then unlocks the deeper implication of this serving for a data product, all abstracts of the platform can be consumed in a self-service manner from a series of predefined assets. Let’s think about this serving more like an internal marketplace or catalogue of assets for delivering everything the data product needs to enable a new node within the wider data mesh.

Read on for some deep thoughts on the topic.

Comments closed

A Free Power BI Sandbox

Published 2022-01-31 by Kevin Feasel

Reza Rad has the right price in mind:

A question I often get from many students is: “How can I practice Power BI service features if I do not have a Power BI Account?”. Not having a Power BI account can happen because of many scenarios; your company might close this option so that the process be only channeled through a specific process within the company. Or you may not have the permission to do so. Not having an account makes it difficult to practice Power BI Service options such as workspace, datasets, dashboards, dataflows, apps, and many other features. On the other hand, even if you have the Power BI Service account, in most of the organizations, you are not the service administrator, so you cannot practice tenant-settings configurations in the service.

Fortunately, there is a way to create your own Power BI sandbox; which means an environment just for yourself, with 25 accounts. You will be the administrator of your environment. The environment will be up for at least 90 days, and you can practice whatever you want for the Power BI service there. The best of all, it is FREE. You don’t have to pay a cent for it. Credit card detail is not needed. What better you could wish for?

Read on to see how.

Comments closed

Multivariate Time Series Anomaly Detection in Azure

Published 2022-01-28 by Kevin Feasel

Louise Han announces an update to the anomaly detection service:

We are excited to announce that we are adding more powerful capabilities in Microsoft Azure Multivariate Anomaly Detector (MVAD) today. In the latest version(v1.1-preview.1) of this API, we implemented a new , in a synchronous manner, which means you could get the anomaly detection results immediately once you call this API. This synchronous inference API is a substantial change compared with previous inference process and will be more intuitive and easier-to-use.

Also, we added a new field named ‘interpretation‘  to give more explanations on an anomaly, like which variables have huge correlation changes and cause the anomaly. These updates will support you to better leverage MVAD and get more useful information to analyze and take actions.

Click through for some more details.

Comments closed

Go/No-Go Indicators for Oracle Migrations to Azure

Published 2022-01-27 by Kevin Feasel

Kellyn Pot’vin-Gorman lays out some guidance on Oracle to Azure migrations:

When migrating an Oracle database to another platform, there are the common indicators and discussion topics around PL/SQL conversions, data types, application rewrites, etc., as being roadblocks to refactoring, but being successful also has to do with the SIZE of the workload coming from Oracle.  I find this is often dismissed, even though this is one of the quickest ways to identify if an ENTIRE Oracle database, (not even by schema or a subset of the Oracle database) can run on a Platform as a Service, (PaaS) solution.

Click through for more information on PaaS limits for Oracle databases in Azure.

Comments closed

Using the Azure Form Recognizer

Published 2022-01-26 by Kevin Feasel

Cem Ayberkin shows off the Azure Form Recognizer:

Shopping malls are facing strong competition and effective loyalty programs boost customer retention. The primary goal of the loyalty scheme is to promote loyalty at the mall, increase footfall whilst understanding shopping habits. With large number of stores and various receipt formats in a mall, the process of manual checking and verification of the data submitted in place did enable rewards to be issued, but proved slow, expensive, inconsistent, and non-scalable. It did not include the valuable line item/product information the mall needed to understand the shopping habits. Therefore, one of the largest shopping malls used Azure Form Recognizer automating receipt scanning and data extraction and feeding the data as rewards points into the customer’s loyalty program, which greatly improved customer shopping experience.

I was pleasantly surprised with how the Form Recognizer works. It’s not perfect but it is useful.

Comments closed

Automating Pipeline Migration to Synapse via Azure DevOps

Published 2022-01-26 by Kevin Feasel

Kevin Chant deploys some Synapse pipelines:

In this post I want to cover how you can automate a pipeline migration to a Synapse workspace using Azure DevOps. As a follow up to a previous post I did about one way to copy an Azure Data Factory pipeline to Synapse Studio.

Because even though the post is good it deserves a follow up showing an automated way of doing it. I wanted to show that it can be done more gracefully.

And we all want to be graceful, right?

Comments closed

Databricks Delta Sharing for Azure

Published 2022-01-25 by Kevin Feasel

Will Girten, et al, announce Delta Sharing on Azure:

Included in this release is a new and improved API for listing all the tables under all schemas in a share. The new API supports pagination similar to other APIs.

For example, to list all the tables in the Delta share my_share, you can simply send a GET request to the /shares/{share_name}/all-tables endpoint on the sharing server.

Prior to that, you might want to read up on Delta Sharing.

Comments closed