Press "Enter" to skip to content

Category: Cloud

Elastic Jobs in Azure SQL DB Now GA

Published 2024-04-15 by Kevin Feasel

Srini Acharya makes an announcement:

Elastic Jobs is a fully integrated Azure SQL database service that allows you to automate and manage administrative tasks across multiple SQL databases in a secure, scalable way. It can run one or more T-SQL job scripts in parallel using Azure portal, PowerShell, REST, or T-SQL APIs. Jobs can be run on a schedule or on-demand, targeting any tier of Azure SQL Database. Job target can include all databases in a server, in an elastic pool, across multiple servers and even databases across different subscriptions and geo regions on Azure. Servers and pools are dynamically enumerated at runtime, so jobs run against all databases that exist in the target group at the time of execution.

If you’ve held off on Azure SQL DB because of a lack of the SQL Agent, take a look at this option.

Comments closed

Speeding up Databricks Lakehouse Queries with Redis

Published 2024-04-11 by Kevin Feasel

Drew Furgiuele has the need for speed:

Since compute and storage are now separated, this means that any time you want to work with your data, you need some form of compute engine that is capable of connecting to and reading your data from your storage locations. Compute engines vary, but one of the best is Apache Spark, which gives you a great distributed compute layer suitable for all sorts of workloads, whether they be analytical and ad-hoc queries, dashboard or BI workloads, data engineering related, or even data science or AI/ML use cases. It really can do it all, and it does it very well.

But what about use operational use cases? For instance: let’s say your Lakehouse is hosting some data that is critical to customer-facing systems that demand low-latency response times, such as real-time users lookups, API interfaces, or event-driven systems, sometimes the overhead required to take a query, schedule it, and run it can be in the hundreds of milliseconds. For some workloads, that’s a lifetime.

Read on to see how you can build a caching layer on top of certain lakehouse operations when some operation needs to be as fast as possible.

Comments closed

Searching for Files in a Blob Storage Container

Published 2024-04-10 by Kevin Feasel

Andy Brownsword hits one of my bugbears:

Shifting from handling data on premises to Azure has been a real change of mindset. Whilst what I want to build may be similar, the how part is completely different. There’s a learning curve not just to the tooling but how you use it too.

This is one of those instances.

I had a storage container with files which had a date in their name. I wanted to perform a wildcard search to select some of them. That sounds straight forward, right?

This is unnecessarily painful, especially if you’re trying to find the right full backup in a container filled with full and transaction log backup files. Andy’s solution does work but also requires a full scan of keys. And I don’t think there’s a better way to do it.

Comments closed

The Importance of Orchestration in E(L)TL Processes

Published 2024-04-10 by Kevin Feasel

Martin Schoombee begins a new series:

In the context of what we’re talking about throughout this series – facilitating the execution of an ETL process in a platform like Azure Data Factory – orchestration means that we’re using the ETL tool primarily for the “E” (Extract) part of the process. In addition to that, most people I know would also use the ETL tool to facilitate the workflow, in other words the order of execution and any constraints that go along with that.

In what I’d like to call the “traditional” approach for lack of a better term, all parts of the ETL process are performed natively by the tool (image below), using whatever built-in tasks are available and of course accounting for any nuances. With this approach, transformations are typically performed in transit and in memory.

Read on to see how the Orchestration approach differs from the traditional ETL approach.

Comments closed

Trying out Microsoft Fabric Mirroring of Cosmos DB

Published 2024-04-10 by Kevin Feasel

Kevin Chant gives it the ol’ college try:

In this post I cover some initial testing of Mirroring Azure Cosmos DB Databases in Microsoft Fabric that I performed.

I wanted to do this post for various reasons. Including the fact that it was announced during the Microsoft Fabric Community Conference that Mirroring is now in Public Preview.

Which means that you can now mirror data from Azure SQL Database, Azure Cosmos DB and Snowflake into your own Microsoft Fabric tenant. Even trial tenants.

Kevin takes us through the process and gives it a try, sharing with us the results of some testing, including a test insertion of 100 million rows.

Comments closed

Cloud Governance Guidance in the Cloud Adoption Framework

Published 2024-04-09 by Kevin Feasel

Stephen Sumner notes an addition to the Microsoft Cloud Adoption Framework (CAF) for Azure:

We are thrilled to announce the latest enhancement to Microsoft’s Cloud Adoption Framework for Azure. We comprehensively updated our cloud governance guidance in the Govern section of the Cloud Adoption Framework (CAF). The updated governance guidance represents Microsoft’s commitment to supporting your organization’s cloud journey, offering a clearer, more accessible, and comprehensive path to effective cloud governance. It encompasses identity, cost, resource, data, and AI governance among other areas of governance categories.

Whether you’re a startup looking to scale efficiently or a large enterprise aiming to refine your governance practices, we designed this governance guidance to meet your needs and guide you to where you need to be.

Read on to learn more about what cloud governance means and the tooling available.

Comments closed

Failover Groups in Azure SQL Database

Published 2024-04-08 by Kevin Feasel

Josephine Bush sets up a failover group in Azure SQL Database:

In today’s fast-paced digital world, keeping your data safe and accessible is more important than ever. That’s where Azure SQL Database steps in, offering a suite of tools to ensure your information is always within reach. Among these tools, failover groups shine, ready to jump into action whenever there’s a hiccup. So, let’s explore how failover groups in Azure SQL Database work their magic, ensuring your data stays safe and sound despite unexpected challenges.

Read on for the step-by-step instructions.

Comments closed

Displaying Azure Maps within a Power BI Paginated Report

Published 2024-04-08 by Kevin Feasel

Chris Webb shows a map in a paginated report:

The built-in mapping functionality in Power BI paginated reports is fairly basic. However the integration of Power Query into Power BI paginated reports gives you an interesting new way of creating maps in paginated reports: you can call the Azure Maps API using Power Query and display the image returned in an Image report item. In this blog post I’ll show you how.

Click through for the demo.

Comments closed

Infrastructure as Code in GitHub

Published 2024-04-04 by Kevin Feasel

I have a new video:

In this video, we look at how to perform Infrastructure as Code in GitHub. We take a Bicep script and generate new Azure resources using it and GitHub Actions.

The video includes a very brief primer on Azure Resource Manager (ARM) and Bicep, and then gets into how you can use GitHub Actions to keep your Azure resources configured the way you expect.

Comments closed

Date Calculation Bug in Power Query ODBC Code

Published 2024-04-03 by Kevin Feasel

Meagan Longoria files a report:

I was working on an imported Power BI semantic model, adding some fiscal year calculations to my date table. The date table was sourced from a view in Databricks Unity Catalog. I didn’t have access to add more fields to the view, so I was adding the fields in Power Query first, with plans to request they be added to the view in the future. I got some unexpected results, which turned into a bug being logged for the ODBC code for Power Query.

If you are only analyzing data in the last 20 years, you won’t see this bug. But if you are doing long-term analysis including years before 2000, you might just run into it.

Read on to see the bug, how you can replicate it, and three workarounds you can use to avoid it.

Comments closed