Press "Enter" to skip to content

Category: Deployment

A Primer on dbt against DuckDB

Robin Moffatt shares a tutorial on dbt:

In 2022 I made a couple of attempts to learn dbt, but it never really ‘clicked’.

I’m rather delighted to say that as of today, dbt has definitely ‘clicked’. How do I know? Because not only can I explain what I’ve built, but I’ve even had the 💡 lightbulb-above-the-head moment seeing it in action and how elegant the code used to build pipelines with dbt can be.

In this blog post I’m going to show off what I built with dbt, contrasting it to my previous hand-built method.

I also had heard of dbt but haven’t really spent the time to learn it because I’m not really a data engineer. But this tutorial has me interested in diving in further.

Leave a Comment

Official Support for fabric-cicd Tool

Yaron Pri Gal announces support for a library:

Today, we’re announcing that fabric‑cicd—the open‑source Python deployment library for Microsoft Fabric—is now an officially supported, Microsoft‑backed tool for CI/CD automation across Fabric workspaces.

Over the past year, fabric‑cicd has rapidly evolved through collaboration with engineering, CAT, MVPs, enterprise customers, and the community. Growing usage, strong sentiment across internal and external channels, and adoption by organizations building enterprise‑grade deployment pipelines helped solidify its value within the Fabric ecosystem.

Read on to learn what this means.

Leave a Comment

Modifying an Azure SQL Database with Mirroring to Fabric Enabled

Olivier Van Steenlandt runs into an issue:

Over the past few weeks, I have been doing some experimenting with Azure SQL mirroring to Microsoft Fabric. In the process, I ran into a couple of issues and challenges. In this data recipe, I will be going through one of my challenges when I got Azure SQL mirroring to Microsoft Fabric setup and running.

At first, everything seemed to be working as expected, and the integration felt very smooth. At that point, I continued to develop my test database in Azure SQL to learn a bit more about mirroring. I made a couple of minor changes to my test database and tried to publish them from my SQL Database Project.

Read on for the issue, as well as the solution—that is, assuming you don’t actually want to change any of the things you’re actively moving over into Fabric.

Leave a Comment

A Quick Explanation of Terraform and Ansible

Amy Abel explains a pair of technologies:

Terraform and Ansible are often mentioned in tech conversations, but DBAs, product teams, and even support staff may not always know what they are or why they matter. Here is a simple way to understand them.

Amy uses the analogy of the crew setting up for a concert. The post doesn’t get into the details of how either technology works, but it does help explain for an outsider why they are interesting technologies to use.

Comments closed

Connecting Microsoft Fabric to Azure DevOps via Service Principal

Yaron Pri Gal doesn’t need no steenkin’ passwords:

Following Azure DevOps Service Principal & Cross Tenant Support (Generally Available) announcement for service principal and cross-tenant support – Microsoft Fabric Git Integration with Azure DevOps (ADO), this blog post serves as a guide to connecting Fabric workspaces to Azure DevOps repositories using service principal.

Fabric Git Integration is the foundation for organizations implementing fully automated CI/CD pipelines, enabling seamless movement of assets across Development, Test, and Production environments.

Currently, Fabric Git Integration supports two major Git providers: Azure DevOps and GitHub. This blog post addresses the new service principal capability for Azure DevOps.

Click through for more info and a link to Microsoft Learn that contains the instructions.

Comments closed

The Good and Bad of Microsoft Fabric Variable Libraries

Jon Lunn digs in:

One of the big issues with Deployment Pipelines in Fabric, or as I call them Disappointment Pipelines, has been the lack of being able to parameterise connections. You do have deployment rules in the pipelines, but they are limited in functionality and don’t support pipeline parameters (boo!), so if you need to push and change items between workspaces in a typical Development, Test and Production workspaces scenario, you had to configure the connections manually, which is a massive pain. Variable Libraries should make the experience of deployment a lot easier.

Read on to see how they work, as well as some of the existing pain points around them.

Comments closed

DACPAC Deployment Misleading Entra ID Error

Koen Verbeeck troubleshoots an issue:

While deploying a DACPAC (from a SQL Server Data Tools Database Project) through Azure Devops, I got the following error message:

The user attempting to perform this operation does not have permission as it is currently logged in as a member of an Azure Active Directory (AAD) group but does not have an associated database user account. A user account is necessary when creating an object to assign ownership of that object. To resolve this error, either create an Azure AD user from external provider, or alter the AAD group to assign the DEFAULT_SCHEMA as dbo, then rerun the statement.

Read on to see what happened in Koen’s case. This is an example of a generic error message hiding relevant information from us.

Comments closed

CI/CD Pipelines with DACPACS Allowing Potential Data Loss

Jess Pomfret describes a use case:

I recently made a change to a database schema that involved removing a column from a table. After we recently improved our authentication process to this tool we no longer needed a password field (don’t worry it was not holding plain text passwords!).

I made the change to the Users.sql file, built the project to confirm everything was good, and committed the change. But, on deployment my pipeline failed and this was the error The schema update is terminating because data loss might occur.

Read on to see how Jess dealt with this scenario.

Comments closed

Creating a Python Package via Poetry

Osheen MacOscar builds a package:

In this blog series (this and the next blog) I am going to demonstrate how to use Poetry to create a Python package, set up testing infrastructure and install it. I am going to be creating a wrapper around the Fantasy Premier League API and creating a function which can create a weekly league table.

This is a straightforward example of how to create a new Python package and add a function call to it.

Comments closed