Press "Enter" to skip to content

Category: DevOps

Creating Fabric Linked Service Parameters for ADO Deployment

Koen Verbeeck glues together several technologies:

Quite the title, so let me set the stage first. You have an Azure Data Factory instance (or Azure Synapse Pipelines) and you have a couple of linked services that point to Fabric artifacts such as a lakehouse or a warehouse. You want to deploy your ADF instance with an Azure Devops build/release pipeline to another environment (e.g. acceptance or production) and this means the linked services need to change as well because in those environments the lakehouse or warehouse are in a different workspace (and also have different object Ids).

When you want to deploy ADF, you typically use the ARM template that ADF automatically creates when you publish (when your instance is linked with a git repo). More information about this setup can be found in the documentation. To parameterize certain properties of a linked service, you can use custom parameterization of the ARM template. Anyway, long story short, I tried to parameterize the properties of the Fabric linked service. 

Read on to see how that went, as well as what you need to do to solve this issue.

Leave a Comment

DevOps in Microsoft Fabric

Hamish Watson lays out what DevOps means in the context of Microsoft Fabric:

Microsoft Fabric (not to be confused with the more general term “fabric” in DevOps) is an integrated data and analytics platform designed for modern data-driven workloads, such as data engineering, business intelligence, and machine learning. With the introduction of Git integration in Microsoft Fabric, DevOps practices are becoming more accessible in the platform, allowing teams to implement collaborative, automated workflows that are common in DevOps environments.

Read on for some of the high-level concepts of what we do with DevOps and how they apply directly to Microsoft Fabric workspaces.

Leave a Comment

CI/CD Pipelines with DACPACS Allowing Potential Data Loss

Jess Pomfret describes a use case:

I recently made a change to a database schema that involved removing a column from a table. After we recently improved our authentication process to this tool we no longer needed a password field (don’t worry it was not holding plain text passwords!).

I made the change to the Users.sql file, built the project to confirm everything was good, and committed the change. But, on deployment my pipeline failed and this was the error The schema update is terminating because data loss might occur.

Read on to see how Jess dealt with this scenario.

Comments closed

Trying out fabric-cicd

Kevin Chant tries a Python package:

In this post I want to cover my initial tests of fabric-cicd. In order to provide some tips for those looking to work with this new offering.

Just so that everybody is aware, fabric-cicd is a Python library that allows you to perform CI/CD of various Microsoft Fabric items into Microsoft Fabric workspaces. At this moment in time there is a limited number of supported item types. However, that list is increasing.

Read on for the test. It currently supports a limit amount of functionality, but it looks promising.

Comments closed

Automating tSQLt Test Execution via Azure DevOps Pipeline

Olivier Van Steenlandt automates some tests:

Before we dive into “how to automate”, I would like to explain why bothering about Unit Test Automation. Unit Tests in general are a good practice to identify potential issues with new source code upfront. Therefore, executing Unit Tests is a good idea. By automating Unit Tests, developers can be sure that Unit Tests will be executed for every single deployment that happens.

In the upcoming step-by-step guide, Unit Test automation will be implemented in a YAML pipeline. The pipelines I’m using are very simplified in comparison with many other pipelines I have seen in the past.

Read on for the process.

Comments closed

A Mechanism for Change Management for DBAs

Terri Hurley shares one path:

Back in the day when I received an email from a developer who requested a Production Change, I would copy and paste SQL code from an email to SQL Server to run against a production database. I never bothered to save that code. But later when I was asked for a history of database changes, it was awkward to locate all the previous email requests asking for production changes. Fast forward to present day, most businesses have embraced Change Management Separation of Duties in companies of all sizes.

My preferred path to change management is:

  • All changes go into source control. You can either have the people making changes create the actual scripts to run, or generate them from base changes in source control, dealer’s choice.
  • Each changeset goes to a specific environment when it gets PR’d into the appropriate branch—that is, dev, QA, test, pre-prod, prod, post-prod, next-gen-post-dev-unprod, whatever.
  • DBAs are on pull requests, at least after a certain point. PR to dev? Let the team handle that code review. Once you move nearer to prod, at least one DBA performs a code review.
  • After PR gating happens, your CI/CD system automatically deploys the changes, assuming it passes any continuous integration tests.

This takes time to get right, but it’s very much worth the effort. I worked at a company where they built the release management processes on top of Git + Azure DevOps. You can also use tools like Flyway.

Comments closed

Continuous Deployment in Practice

Bravin Wasike completes a series on continuous integration and continuous deployment:

Continuous Deployment (CD) is an advanced CI/CD practice that takes automation to the next level. It automatically deploys every code change that passes the automated testing phase to production. Unlike Continuous Delivery, which requires manual approval before deploying to production, Continuous Deployment eliminates manual intervention. This allows for seamless and frequent software updates.

Click through for the full article.

Comments closed

Running a Microsoft Fabric Notebook via Azure DevOps

Kevin Chant runs a notebook:

In this post I want to share one way that you can run a Microsoft Fabric notebook from Azure DevOps.

You can consider this post a follow-up to my last post about unit tests on Microsoft Fabric items. Since somebody asked me about automating notebooks and I wanted to show it in action.

Please note, currently the ability to call the API that runs a notebook on demand does not support service principals.

Despite that limitation, Kevin shows two ways to authenticate while calling the appropriate API.

Comments closed

CI/CD in GitHub

I have a new video:

In this video, I explain what continuous integration (CI) is, disambiguate continuous delivery from continuous deployment (CD), and see how you can perform CI/CD operations using GitHub Actions.

Read on to see what these terms mean and an example of how it all works with .NET projects.

Comments closed