Press "Enter" to skip to content

Category: Cloud

Using Query Store on Azure Database for PostgreSQL

Published 2024-07-24 by Kevin Feasel

Marisa Mathews covers an episode of Data Exposed with Anna Hoffman and Grant Fritchey:

Capturing query metrics in PostgreSQL can be a challenge. Add in running that PostgreSQL database on Azure and things get even more challenging. However, Microsoft has a great addition to PostgreSQL when running in Azure Database, Query Store. This session will show the great query information available within the Query Store and how to query it.

Click through for a link to the video and additional links.

Comments closed

Calling a REST Endpoint from Azure SQL Database

Published 2024-07-23 by Kevin Feasel

Meagan Longoria makes a call:

External REST endpoint invocation in Azure SQL DB went GA in August 2023. Whereas before, we might have needed an intermediate technology to make a REST call to an Azure service, we can now use an Azure SQL Database to call a REST endpoint directly.

One use case for this would be to retrieve a file from blob storage. I explain how to set this up below.

Read on to learn more about the process, as well as a few ideas on when you might use it.

Comments closed

Creating Elastic Jobs in TerraForm

Published 2024-07-23 by Kevin Feasel

Josephine Bush builds a script:

If you aren’t familiar with Terraform and its requisite bits and pieces, I have a post that walks you through them. Let’s hit the ground running. To create Elastic Jobs with Terraform, you need to use the azapi_resource, but before we go down that path, you will use the azurerm resource to create some supporting objects.

Read on for a guide covering requirements and setup.

Comments closed

Choosing between Azure SQL DB Ledger and Azure Confidential Ledger

Published 2024-07-18 by Kevin Feasel

Pieter Vanhove reminds me that ledger tables exist:

Ledger technology is a way of storing data that ensures its integrity, immutability, and verifiability. It can be used for scenarios where trust and transparency are essential, such as financial transactions, supply chain tracking, or regulatory compliance. Azure offers two services that leverage ledger technology to provide tamper-proof data storage: ledger in Azure SQL Database and Azure Confidential Ledger. In this blog post, we will compare these two services and help you decide which one is best suited for your needs.

Read on for the comparison. I think the answer for most cases is “neither” but there are specific times when ledger tables could make a good amount of sense.

Comments closed

Finding Query History in Snowflake

Published 2024-07-17 by Kevin Feasel

Kevin Wilkie digs into the history:

If you’re running an audit process in SQL Server, you can do this fairly easily. But how many of us work in a place that requires this or has enough space to do this? Sadly, very few of us…

If you have Query Store running (and why wouldn’t you?) you can find a specific query you were running. But you have to have a specific something you can search for…

But can you get just a list of what you specifically ran in order? Not that I can find….

My recollection is that, if you’re using Azure SQL Database, that auditing is on by default (or you can turn it on with a toggle switch or radio buttion). For Azure Synapse Analytics dedicated SQL pools, there are a few DMVs that cover query operations. But yeah, in general, it’s something you’d need to enable first.

Comments closed

Script Tasks in Azure DevOps Pipelines

Published 2024-07-17 by Kevin Feasel

Justin Bird builds a script task:

There are several dedicated script tasks that can be used for different purposes, some of the tasks have shortcut tasks which require much less input and are arguably easier to read. In the examples below, I have included the required inputs needed to run the scripts but in most cases, other optional inputs are available.

This looks pretty similar to the way you’d do it in GitHub Actions workflows as well—no surprise there, considering that the same group of people who built pipelines in ADO handled Actions in GitHub.

Comments closed

Tips for Using Azure Backup for SQL Server

Published 2024-07-15 by Kevin Feasel

Anna Hoffman, et al, share some tips and tricks:

We recently worked with a customer that migrated their Windows and SQL Servers to Azure that wanted to use Azure Backup for a consistent enterprise backup experience. The SQL Servers had multiple databases of varying sizes, some that were multi-terabyte.  A single Azure Backup vault was deployed using a policy that was distributed to all the SQL Servers. During the migration process, the customer observed issues with the quality of the backups and poor virtual machine performance while the backups were running. We worked through the issues by reviewing the best practices, modifying the Azure Backup configuration, and changing the virtual machine SKU. For this specific example, the customer needed to change their SKU from Standard_E8bds_v5 to Standard_E16bds_v5 to support the additional IOPS and throughput required for the backups.  They used premium SSD v1 and the configuration met the IOPS and throughput requirements.  

In this post, we share some of the techniques we used to identify and resolve the performance issues that were observed. 

Read on to learn more about how Azure Backup works and troubleshooting mechanisms.

Comments closed

RIP Stretch DB

Published 2024-07-05 by Kevin Feasel

Debbi Lyons calls it:

Ever since Microsoft introduced SQL Server Stretch Database in 2016, our guiding principles for such hybrid data storage solutions have always been affordability, security, and native Azure integration. Customers have indicated that they want to reduce maintenance and storage costs for on-premises data, with options to scale up or down as needed, greater peace of mind from advanced security features such as Always Encrypted and row-level security, and they seek to unlock value from warm and cold data stretched to the cloud using Microsoft Azure analytics services.     

During recent years, Azure has undergone significant evolution, marked by groundbreaking innovations like Microsoft Fabric and Azure Data Lake Storage. As we continue this journey, it remains imperative to keep evolving our approach on hybrid data storage, ensuring optimal empowerment for our SQL Server customers in leveraging the best from Azure.

This is not surprising at all, considering that the premise of Stretch DB was that you could off-load old and less-important data from your local SQL Server instances and expensive local disk into Azure, querying it when you need that data. The problem was, you couldn’t use cheap storage and pay a few cents per gigabyte of data per month. Instead, you were effectively spinning up Azure Synapse Analytics and paying a marked premium for your least important data. The price alone made this an untenable idea, but there were other holes in the plan as well that doomed it as a product.

Comments closed

Parallel Download in Oracle Object Storage

Published 2024-07-03 by Kevin Feasel

Brendan Tierney continues a series on Oracle Object Storage:

In previous posts, I’ve given example Python code (and functions) for processing files into and out of OCI Object and Bucket Storage. One of these previous posts includes code and a demonstration of uploading files to an OCI Bucket using the multiprocessing package in Python.

Building upon these previous examples, the code below will download a Bucket using parallel processing. Like my last example, this code is based on the example code I gave in an earlier post on functions within a Jupyter Notebook.

Click through for the code.

Comments closed

Export Azure SQL DB to Blob Storage

Published 2024-07-03 by Kevin Feasel

Josephine Bush runs an import-export business and wants a database to “fall off a truck”:

After a data migration, we needed to decommission the old Azure SQL DBs, but we wanted to keep a copy in case we needed anything later. Enter exporting an Azure SQL DB to storage!

Click through for an example of how it works. Given that we’re getting bacpac files out, I wonder what it would look like with a really large database.

Comments closed