Press "Enter" to skip to content

Category: Cloud

Right to Be Forgotten in Delta Lake

Published 2022-03-30 by Kevin Feasel

Milos Colic, et al, tackle a tricky problem:

With Delta, we have one more tool at our disposal to address GDPR compliance and, in particular, “the right to be forgotten” – VACUUM. Vacuum operation removes the files that are no longer needed and that are older than a predefined retention period. The default retention period is 30 days to align with GDPR definition of undue delay. Our earlier blog on a similar topic explains in detail how you can find and delete personal information related to a consumer by running two commands:

The part I’m finding tricky here is, how does this handle “time travel” scenarios in which you’re looking at prior iterations of data? I haven’t run through all of the scenarios so this is just speculation, but it seems that even with all of these changes, you’d still have to worry about historical data containing that sensitive information.

Comments closed

Migrating SSIS On-Prem Workloads into Azure

Published 2022-03-30 by Kevin Feasel

Jitendra Yadeo has put together a how-to guide:

– There can be scenario where organization wants to migrate there existing SSIS ETL process on cloud so instead of rewriting SSIS package using Cloud specific ETL tools like Azure Data Factory we can directly migrate SSIS packages and call it through Azure Data Factory.

– Goal of this blog is to show how SSIS packages hosted on on-premise can be migrated to Azure Data Factory (ADF) using Azure-SSIS Integration Runtime (IR).

Read on for a step-by-step guide.

Comments closed

Configuring FIDO2 for Azure Active Directory

Published 2022-03-29 by Kevin Feasel

Joey D’Antoni takes us through a process:

If this sounds scary, and it does to me, who is by far not an expert in all things security, but knows a little bit, you may ask, what are some alternative solutions? The answer to that question is Fido2, a different protocol for MFA and auth. Remember all of that stuff Microsoft talks about with passwordless login? That’s all based around Fido2. I configured this for DCAC’s Azure Active Directory yesterday, and I wanted to walk you through the steps.

Click through to learn how.

Comments closed

Azure ML Well-Architected Framework Review

Published 2022-03-28 by Kevin Feasel

Ben Brauer has good news:

Microsoft offers prescriptive guidance called the Well-Architected Framework that optimizes workloads implemented and deployed on Azure. This guidance has been generalized for most workloads and creates a basis for reliable and secure applications that are cost optimized.

We have begun to build on this base content set to include more precise guidance for specific workload types, such as machine learning, data services and analytics, IoT, SAP, mission critical apps, and web apps. Machine Learning was the first branch from the base content, which came into fruition in the Fall of 2021.

In case you have never used the Azure Well-Architected Review assessment tool, it’s really useful. It can take hours (or days) to go through the review but if you take it seriously and have the right people in the room giving answers, you’ll get concrete guidance on how to optimize your Azure-based solutions.

Comments closed

Sending E-Mail via Azure Logic Apps

Published 2022-03-24 by Kevin Feasel

Maria Zakourdaev replaces Database Mail:

Azure SQL database is a fully managed relational database in the Microsoft cloud. It is a scalable, durable and performant database engine that allows developers to focus on the business logic without worrying about storage or server uptime. The main difference between SQL Server on premises and Azure SQL Database is that in Azure SQL Database only database level features are available. Features, like SQL Server Agent or SQL Server Mail are not supported by Azure SQL Database. However, in some situations we want to send an email from inside stored procedures. This post will show how to solve this challenge.

I’ve found Logic Apps to be surprisingly easy for this kind of thing, including connectors to outside mail services like Gmail. H/T Maria via Madeira Data’s blog.

Comments closed

Azure Purview Workflows in Public Preview

Published 2022-03-24 by Kevin Feasel

Victoria Holt makes note of an improvement to Purview:

Azure Purview Workflows moved to Public Preview 10 March.  This functionality enables customers to orchestrate the create, update and delete operations of data entities, have validation, and approval of these data entities using repeatable business processes.

The benefit of using Purview workflows is higher quality data, policy compliance, user collaboration, and change tracking awareness across the organization.

Read on for more information about how workflows currently work.

Comments closed

Running SQL Scripts on Snowflake from Azure Data Factory

Published 2022-03-22 by Kevin Feasel

Koen Verbeeck shows off the Script activity in Azure Data Factory:

Azure Data Factory has a new activity introduced this week (around the 10th of March 2022 for you future readers): the Script activity! This is not to be confused with the script task/component of SSIS, which allows you to execute .NET script (C# for most people, or VB if you’re Ben Weissman). No, this task executes SQL, so it’s more akin to the Execute SQL Task of SSIS.

Click through to see how it works while I lament the fact that SSIS never supported the best .NET language.

Comments closed

An Overview of Azure Redis Cache

Published 2022-03-22 by Kevin Feasel

Arun Sirpal lays out the use case of Azure Redis Cache:

Redis Cache is a well know caching technology and you can run it in Azure as a fully managed service. A common requirement (the most basic one) is doing a workflow like:

1. When an application needs to retrieve data, it will first search to see if it exists in Azure Cache for Redis.

2. If the data is found in Azure Cache for Redis (cache hit) use it

3. If the data is not found in Azure Cache for Redis (cache miss), then the application will need to retrieve the data from Azure SQL (or whatever cloud db back end you use)

4. For cache miss scenarios, the requesting application should add the data retrieved from the Azure Database to Azure Cache for Redis.

This is also known as the cache-aside pattern. If you’re feeling really cheeky, you can combine cache-aside with the decorator pattern to “hide” the cache in your code.

Comments closed

Managed Instance Link in Preview

Published 2022-03-18 by Kevin Feasel

Dani Ljepava announces support for Managed Instance link is now in public preview:

As of today, we are pleased to announce that the link feature for Managed Instance is available in the open public preview, in all Azure regions worldwide. It can be used with existing, or new managed instances, and SQL Server 2019 Enterprise, or Developer edition, including SQL Server 2022 CTP (available through EAP). We have also released the tooling support for the link in the form of automated wizards available in SQL Server Management Studio, starting from SSMS v18.11.1.

With the link, replicated databases from SQL Server on Managed Instance are usable as R/O secondary replicas. While the link is in operation, transactions commited on SQL Server (primary) are instantaneously committed to Managed Instance (secondary). This provides an exact replica of your SQL Server database on Managed Instance, synced near real-time. The link was built to be resilient, in case of the network being down, SQL Server being rebooted, or maintained, or in case of some other issue, the link will automatically resume replicating where it has left off when the issue has been resolved.

Support for 2019 is a shrewd idea, given the SQL Server version adoption curve for companies. This isn’t going to replace having a proper availability group for high availability or even (most) disaster recovery options, though, because the link is currently one-way—though Dani does mention eventual support for bi-directional operation with SQL Server 2022.

Comments closed