Press "Enter" to skip to content

Category: Cloud

A Reference Architecture for Microsoft Fabric

Published 2024-08-15 by Kevin Feasel

James Serra draws boxes:

Microsoft Fabric uses a data lakehouse architecture, which means it does not use a relational data warehouse (with its relational engine and relational storage) and instead uses only a data lake to store data. Data is stored in Delta lake format so that the data lake acquires relational data warehouse-like features (check out my book that goes into much detail on this, or my video). Here is what a typical architecture looks like when using Fabric (click here for the .vsd):

Click through for the image as well as James’s explanation of the components.

Comments closed

Backing SQL Server up to S3 Buckets

Published 2024-08-15 by Kevin Feasel

David Fowler backs up a database:

Way back in the mists of time I wrote a post on how to backup SQL server to an S3 bucket using TNTDrive, https://sqlundercover.com/2018/06/18/backup-your-on-premise-sql-server-directly-to-an-aws-s3-bucket/.

Back then, if we wanted to backup SQL to S3 we needed to use a third party tool. Since SQL 2022 things have changed and we’ve now got the option to backup directly to S3 in a similar way that we can backup to Azure BLOB store.

And, going one step further, you can also use PolyBase to read data from S3 buckets in SQL Server 2022.

Comments closed

Cosmos DB HTAP into Azure Synapse Analytics and Microsoft Fabric

Published 2024-08-13 by Kevin Feasel

Paul Hernandez doesn’t want to write ETL jobs:

In the ever-evolving landscape of data management and analytics, choosing the right tools and approaches is crucial for optimizing performance and achieving business goals. Two prominent solutions that have gained traction are Azure Synapse Link for Azure Cosmos DB and Mirroring in Microsoft Fabric. Both offer unique benefits and cater to different needs, making it essential to understand their differences and use cases.

Read on to see how each of these works, as well as a quick demonstration of efficacy.

Comments closed

Blob Storage Account Lifecycle Maintenance

Published 2024-08-09 by Kevin Feasel

Andy Brownsword deletes some files but wants to keep other files:

A hierarchy of directories which contain files. That’s how we typically think about file storage. That’s not quite the same everywhere. In Blob Storage a file can appear to be in a directory, but when it’s removed so is the directory.

This can occur when using Lifecycle Management to help purge legacy blobs, which can be unexpected. Let’s look at a way we can help remediate this.

One important thing to remember about Azure blob storage accounts and S3 buckets is that there’s really no concept of a directory structure. It’s all keys, where your key might be dir1/dir2/dir3/file.txt. This is a bit different for Azure Data Lake Storage Gen2 and its notion of hierarchical namespaces (i.e., folders). But Andy does walk through some of the consequences of this and how to work with lifecycle maintenance policies to delete only certain sets of files.

Comments closed

Ad Hoc Data Exploration with Azure Data Explorer

Published 2024-08-01 by Kevin Feasel

Michal Bar introduces a new feature:

We are excited to introduce the new Data Exploration feature, designed to enhance your ability to delve deeper into the data presented on any Dashboard.

If the information you’re seeking isn’t readily available on the dashboard, this feature allows you to extend your exploration beyond the data displayed in the tiles, potentially uncovering new insights.

Directly from a dashboard, you can refine your exploration using a user-friendly, form-like interface. This intuitive and dynamic experience is tailored for insights explorers seeking insights based on high volumes of data in near real time.

Click through to see the new feature in action.

Comments closed

Tracking Task History in Snowflake

Published 2024-08-01 by Kevin Feasel

Kevin Wilkie is interested in just one thing:

By the table function name, you’re probably wondering “Sherpa, why on Earth do I really need to look at query history based on the session? I have Query_History_By_User which gives me a broader look at my data. Who really cares?”

Great question, my friend! You’re right. This new Query_History_By_Session table function isn’t going to give you a lot of data that you can’t get easier – and more helpfully – with Query_History_By_User. Why did Snowflake provide me this “useless” function?

Read on for the value of this function and an example of querying task history.

Comments closed

Working with Managed Entities in Azure SQL DB

Published 2024-07-26 by Kevin Feasel

Josephine Bush creates and uses a managed identity:

Benefits of Using Managed Identities and Entra Groups

  • Enhanced Security: Using managed identities eliminates the need to manage credentials, reducing the risk of credential theft.
  • Simplified Management: Entra Groups streamline the management of permissions for multiple users or managed identities, making it easier to apply consistent access policies.
  • Scalability: As your organization grows, you can easily manage access by adding new users or managed identities to Entra Groups without needing to update database permissions individually.

Read on to see how you can create one and what you can do with it.

Comments closed

Using Query Store on Azure Database for PostgreSQL

Published 2024-07-24 by Kevin Feasel

Marisa Mathews covers an episode of Data Exposed with Anna Hoffman and Grant Fritchey:

Capturing query metrics in PostgreSQL can be a challenge. Add in running that PostgreSQL database on Azure and things get even more challenging. However, Microsoft has a great addition to PostgreSQL when running in Azure Database, Query Store. This session will show the great query information available within the Query Store and how to query it.

Click through for a link to the video and additional links.

Comments closed

Calling a REST Endpoint from Azure SQL Database

Published 2024-07-23 by Kevin Feasel

Meagan Longoria makes a call:

External REST endpoint invocation in Azure SQL DB went GA in August 2023. Whereas before, we might have needed an intermediate technology to make a REST call to an Azure service, we can now use an Azure SQL Database to call a REST endpoint directly.

One use case for this would be to retrieve a file from blob storage. I explain how to set this up below.

Read on to learn more about the process, as well as a few ideas on when you might use it.

Comments closed

Creating Elastic Jobs in TerraForm

Published 2024-07-23 by Kevin Feasel

Josephine Bush builds a script:

If you aren’t familiar with Terraform and its requisite bits and pieces, I have a post that walks you through them. Let’s hit the ground running. To create Elastic Jobs with Terraform, you need to use the azapi_resource, but before we go down that path, you will use the azurerm resource to create some supporting objects.

Read on for a guide covering requirements and setup.

Comments closed