Press "Enter" to skip to content

Category: Cloud

Searching for Files in a Blob Storage Container

Published 2024-04-10 by Kevin Feasel

Andy Brownsword hits one of my bugbears:

Shifting from handling data on premises to Azure has been a real change of mindset. Whilst what I want to build may be similar, the how part is completely different. There’s a learning curve not just to the tooling but how you use it too.

This is one of those instances.

I had a storage container with files which had a date in their name. I wanted to perform a wildcard search to select some of them. That sounds straight forward, right?

This is unnecessarily painful, especially if you’re trying to find the right full backup in a container filled with full and transaction log backup files. Andy’s solution does work but also requires a full scan of keys. And I don’t think there’s a better way to do it.

Leave a Comment

The Importance of Orchestration in E(L)TL Processes

Published 2024-04-10 by Kevin Feasel

Martin Schoombee begins a new series:

In the context of what we’re talking about throughout this series – facilitating the execution of an ETL process in a platform like Azure Data Factory – orchestration means that we’re using the ETL tool primarily for the “E” (Extract) part of the process. In addition to that, most people I know would also use the ETL tool to facilitate the workflow, in other words the order of execution and any constraints that go along with that.

In what I’d like to call the “traditional” approach for lack of a better term, all parts of the ETL process are performed natively by the tool (image below), using whatever built-in tasks are available and of course accounting for any nuances. With this approach, transformations are typically performed in transit and in memory.

Read on to see how the Orchestration approach differs from the traditional ETL approach.

Leave a Comment

Trying out Microsoft Fabric Mirroring of Cosmos DB

Published 2024-04-10 by Kevin Feasel

Kevin Chant gives it the ol’ college try:

In this post I cover some initial testing of Mirroring Azure Cosmos DB Databases in Microsoft Fabric that I performed.

I wanted to do this post for various reasons. Including the fact that it was announced during the Microsoft Fabric Community Conference that Mirroring is now in Public Preview.

Which means that you can now mirror data from Azure SQL Database, Azure Cosmos DB and Snowflake into your own Microsoft Fabric tenant. Even trial tenants.

Kevin takes us through the process and gives it a try, sharing with us the results of some testing, including a test insertion of 100 million rows.

Leave a Comment

Cloud Governance Guidance in the Cloud Adoption Framework

Published 2024-04-09 by Kevin Feasel

Stephen Sumner notes an addition to the Microsoft Cloud Adoption Framework (CAF) for Azure:

We are thrilled to announce the latest enhancement to Microsoft’s Cloud Adoption Framework for Azure. We comprehensively updated our cloud governance guidance in the Govern section of the Cloud Adoption Framework (CAF). The updated governance guidance represents Microsoft’s commitment to supporting your organization’s cloud journey, offering a clearer, more accessible, and comprehensive path to effective cloud governance. It encompasses identity, cost, resource, data, and AI governance among other areas of governance categories.

Whether you’re a startup looking to scale efficiently or a large enterprise aiming to refine your governance practices, we designed this governance guidance to meet your needs and guide you to where you need to be.

Read on to learn more about what cloud governance means and the tooling available.

Leave a Comment

Failover Groups in Azure SQL Database

Published 2024-04-08 by Kevin Feasel

Josephine Bush sets up a failover group in Azure SQL Database:

In today’s fast-paced digital world, keeping your data safe and accessible is more important than ever. That’s where Azure SQL Database steps in, offering a suite of tools to ensure your information is always within reach. Among these tools, failover groups shine, ready to jump into action whenever there’s a hiccup. So, let’s explore how failover groups in Azure SQL Database work their magic, ensuring your data stays safe and sound despite unexpected challenges.

Read on for the step-by-step instructions.

Leave a Comment

Displaying Azure Maps within a Power BI Paginated Report

Published 2024-04-08 by Kevin Feasel

Chris Webb shows a map in a paginated report:

The built-in mapping functionality in Power BI paginated reports is fairly basic. However the integration of Power Query into Power BI paginated reports gives you an interesting new way of creating maps in paginated reports: you can call the Azure Maps API using Power Query and display the image returned in an Image report item. In this blog post I’ll show you how.

Click through for the demo.

Leave a Comment

Infrastructure as Code in GitHub

Published 2024-04-04 by Kevin Feasel

I have a new video:

In this video, we look at how to perform Infrastructure as Code in GitHub. We take a Bicep script and generate new Azure resources using it and GitHub Actions.

The video includes a very brief primer on Azure Resource Manager (ARM) and Bicep, and then gets into how you can use GitHub Actions to keep your Azure resources configured the way you expect.

Leave a Comment

Date Calculation Bug in Power Query ODBC Code

Published 2024-04-03 by Kevin Feasel

Meagan Longoria files a report:

I was working on an imported Power BI semantic model, adding some fiscal year calculations to my date table. The date table was sourced from a view in Databricks Unity Catalog. I didn’t have access to add more fields to the view, so I was adding the fields in Power Query first, with plans to request they be added to the view in the future. I got some unexpected results, which turned into a bug being logged for the ODBC code for Power Query.

If you are only analyzing data in the last 20 years, you won’t see this bug. But if you are doing long-term analysis including years before 2000, you might just run into it.

Read on to see the bug, how you can replicate it, and three workarounds you can use to avoid it.

Comments closed

Maintaining Dynamic IP Rules for Azure Network Security Groups

Published 2024-04-02 by Kevin Feasel

Daniel Hutmacher shares a couple scripts:

Recently, my home ISP has started changing my public IP address. This causes me some headache because I have a couple of Azure Network Security Group rules (think of them as firewall rules) that specifically allow my home IP access to all of my Azure resources. When my home IP changes, those rules have to be updated accordingly.

So I made a PowerShell-based solution to automatically maintain them.

Read on for the process.

Comments closed

Using Azure SQL Database Elastic Pools

Published 2024-04-01 by Kevin Feasel

Josephine Bush shares some tips on using elastic pools in Azure SQL DB:

Main points about elastic pools

  • Elastic pools enable you to purchase resources for a pool shared by multiple databases to accommodate unpredictable usage periods by individual databases.
  • You can create multiple pools on a server, but you can’t add databases from different servers into the same pool.
  • Pools are well-suited for many databases with specific utilization patterns. These patterns are characterized by low average utilization with infrequent utilization spikes for a given database. Conversely, multiple databases with persistent medium-high utilization shouldn’t be placed in the same elastic pool.

All the databases in the elastic pool share the DTU and storage. This is very apparent when multiple persistent medium-high utilization databases are in the same pool, which is why Microsoft advises against it.

Read on to see information about storage, DTUs, noisy neighbors, and more.

Comments closed