Press "Enter" to skip to content

Category: Cloud

Synapse and Azure ML Pipelines

Published 2023-02-23 by Kevin Feasel

Santosh Thomas integrates two Azure products:

As more customers standardize on the Synapse data platform, enabling machine learning workflows through Azure Machine Learning (Azure ML) becomes particularly interesting. This is especially true as more customers look to bring their data engineering and data science practices together and mature capabilities on both sides.

The goal of this blog post is to highlight how Synapse and Azure ML can work well together to deliver key insights. This is motivated by a scenario where a customer modernized their data platform on Azure Synapse but was looking to improve their data science practices through Azure ML. The focus of this blog is to expose existing functionality, and it is not a “hardened” solution with security or other cloud best practice implementations. The workflow steps also assume some level of comfort with Python and working with the Azure Python SDKs.

There was a time in which Microsoft wanted us to remain in Synapse for machine learning tasks, but that time is gone: the emphasis is definitely to do machine learning tasks in Azure ML, regardless of where the data lives…unless there’s a Spark job involved, in which case things get all weird again.

Comments closed

Kafka Control and Data Planes

Published 2023-02-22 by Kevin Feasel

Sanjay Garde explains how the architecture of Apache Kafka solutions has expanded over time:

With the advent of service mesh and containerized applications, the idea of the control and data plane has become popular. A part of your application infrastructure, such as a proxy or sidecar, is dedicated to aspects such controlling traffic, access, governance, security, and monitoring and is referred to as the control plane. Another part of your application infrastructure that is used purely for processing your business transactions is referred to as the data plane.

Read on to see how the concept works at an architectural level.

Comments closed

ADX Dashboards Now Generally Available

Published 2023-02-22 by Kevin Feasel

Michal Bar provides an overview of Azure Data Explorer functionality now generally available :

Each ADX dashboard is a collection of tiles, optionally organized in pages, where each tile has an underlying query and a visual representation. Using the web UI, you can natively export Kusto Query Language (KQL) queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed. In addition to ease of data exploration, this fully integrated Azure Data Explorer dashboard experience provides improved query and visualization performance.

Read on to learn more.

Comments closed

Delta Lake Support in Azure Stream Analytics

Published 2023-02-20 by Kevin Feasel

Emma An makes an announcement:

Delta Lake has gained popularity in recent times due to its unique features and advantages over traditional data warehouse and other storage formats. For those already using traditional data storage format or moving to a lakehouse architecture, Delta Lake can offer several compelling benefits that can further enhance the performance and capabilities of their data pipelines. Many Azure services are integrated with Delta Lake, and now you can use Azure Stream Analytics to write in Delta format.

In this blog, we will explain the native support of Delta Lake in Azure Stream Analytics, that can help users take their workload to the next level, providing a seamless and scalable solution for large-scale data processing and storage. It is easy to start, taking only a few clicks to create an end-to-end pipeline, and write to either a new or existing Delta table stored in Azure Data Lake Storage Gen2.

This is a nice addition to Stream Analytics and Emma shows two ways you can write out results in Delta Lake format.

Comments closed

First Thoughts on Azure Hyperscale Serverless

Published 2023-02-20 by Kevin Feasel

Reitse Eskens shares some thoughts:

As some of you know, I’ve written a series of blog posts on Azure SQL Databases and there’s an accompanying session that I had the honour of presenting a number of times.
Now Azure keeps developing new offers and one of these went in public preview February 15th. An offer I hadn’t seen coming. You can read the introductory post here.

It’s the Azure Hyperscale Serverless option.

Read on for Reitse’s impressions from the preview. This wasn’t a torture test but did provide an overview of how to create and load data into the database. Reitse also calculates the cutoff point when you should switch from Serverless to traditional Hyperscale, so check that out as well.

Comments closed

Building an Internal Load Balancer in Azure

Published 2023-02-17 by Kevin Feasel

Vaibhav Kumar balances the scales:

The Internal load balancer manages load for a private network with any inbound access from the public platform. As in the diagram below, the primary load balancer managing load from the internet is a public-type load balancer. But, the VMs communication to storage or database is managed through a type-internal load balancer.

Click through for a walkthrough of the process.

Comments closed

Working with Postgres Extensions in Azure Cosmos DB

Published 2023-02-17 by Kevin Feasel

Sarah Dutkiewicz runs into an issue:

Problem: I installed PostGIS on my single-node cluster without issues. However, I scaled my cluster to 2 nodes afterwards. When I ran the query that uses ST_X and ST_Y from PostGIS, I got the following error:

ERROR:  type "public.geometry" does not exist
CONTEXT:  while executing command on private-w0.azure-cosmos-db-global-ug-demo.postgres.database.azure.com:5432

When I read the CONTEXT message, I realized by the w# reference that the worker nodes didn’t have PostGIS installed. When you scale the nodes – at least in this case, it doesn’t enable the extensions over there.

Read on to see how Sarah was able to resolve this issue.

Comments closed

Azure Defender for SQL Overview

Published 2023-02-16 by Kevin Feasel

Deepthi Goguri looks at an Azure security offering:

Azure Defender for SQL, once you enable it will alert you for any SQL injection attacks, brute force attacks or any breached identities trying to access the data of your database. It also provides the vulnerability assessments. Vulnerability assessments give you alerts about the configurations of your database. If your database configuration is not following the standards of Azure, you will receive the alerts in the vulnerability assessment report.

You can enable the Azure Defender at the subscription level or at the Server level or at the resource level as well. Under the recommendations in the security center in the Azure portal, check for the Remediate security configuration. This will show if the Azure defender is configured properly.

I like Azure Defender for SQL, especially the advanced threat protection element. It’s based on IP address location and has caught me in different locations as I’ve traveled.

Comments closed

Azure ML Overview

Published 2023-02-16 by Kevin Feasel

Sanil Mhatre gets us started with Azure Machine Learning:

The five-part series is designed to jump-start any IT professional’s journey in the fascinating world of Data Science with Azure Machine Learning (Azure ML). Readers don’t need prior knowledge of Data Science, Machine Learning, Statistics, or Azure to begin this adventure.

All you will need is an Azure subscription and I will show you how to get a free one that you can use to explore some of Azure’s features before I show you how to set up the Azure ML environment.

Part 1 is available now, with the other parts coming up soon. Even so, Part 1 is a big article on its own.

Comments closed

Distributing Azure Costs by Cost Centers, Divisions, and Projects

Published 2023-02-14 by Kevin Feasel

Pranab Paul divvies up the costs:

While working on various customer and partner facing roles, I felt the necessity of a simple and flexible solution to align Azure Cost to the customer’s organizational structure. “Project Bose” is a fully operational prototype derived from the same thought process. This is a side project I am working on during my leisure time. I found various customers derived similar solutions in-house, and there are ISV solutions as well. But there are a few fundamental differences between “Project Bose” and all the other solutions I found. “Project Bose” has a flexible backend and hence any changes in organizational structure can easily be implemented on it without disruption. It is also independent of using Resource Tags, which gives it the opportunity to remain non-vulnerable to erroneous values injected intentionally or non-intentionally by IT-Ops.

Pranab didn’t include a link to the GitHub repo, so here it is.

Comments closed