A Fine Slice Of SQL Server
This post covers how to update both ends of Azure Synapse Link for SQL Server 2022 using Azure DevOps. As shown at the Data Toboggan conference.
By the end of this post you will know how to deploy database updates to both the SQL Server database and the Azure Synapse dedicated SQL Pool that are used as part of Azure Synapse Link for SQL Server 2022, using a pipeline in Azure DevOps. To keep them consistent.
Click through for the process.
Comments closedSarah Dutkiewicz compares and contrasts:
Every time I do an Azure DevOps talk, I get someone asking me about migrating from GitHub to Azure DevOps. Every time, I have to ask “Why do you want to migrate from GitHub to Azure DevOps?” Why would you choose between Azure DevOps and GitHub? Or better yet – do you have to choose between them? Let’s look at how they compare and the tooling available.
This is a really tough question and Sarah helps explain why.
Comments closedEitan Blumin asks and answers a question:
Following the rise in popularity of DevOps for Databases, many interesting questions are being asked on the topic.
One of these questions is: Should your SQL Database project be in the same source control repository and solution as the App code project? Or maybe they should be in the same repository but separate solutions? Or maybe they should be in completely separate repositories?
Pre-registering my answer here: for most organizations, databases should be in a separate repository. The deployment cadence is different, the deployment mechanism is different, and the people working on each likely differ. Read on for Eitan’s thoughts, which get into more of the nuance behind the answer.
Comments closedVaibhav Sethi and Roland Faeustlin make an announcement:
We are happy to announce native support for Git in Databricks Workflows, which enables our customers to build reliable production data and ML workflows using modern software engineering best practices. Customers can now use a remote Git reference as the source for tasks that make up a Databricks Workflow, for example, a notebook from the main branch of a repository on GitHub can be used in a notebook task. By using Git as the source of truth, customers eliminate the risk of accidental edits to production code. They also remove the overhead of maintaining a production copy of the code in Databricks and keeping it updated, and improve reproducibility as each job run is tied to a commit hash. Git support for Workflows is available in Public Preview and works with a wide range of Databricks supported Git providers including GitHub, Gitlab, Bitbucket, Azure Devops and AWS CodeCommit.
Read on to see how it works.
Comments closedTori Tompkins explains what MLOps is and why it’s valuable:
A ML project will typically begin in an ‘Explore Phase’ where a data scientist or team of data scientists will explore the data they currently have and experiment with models, algorithms, parameters and features. MLOps at this stage is responsible for supplying Data Scientists with environment they need to achieve this. One way this can be done is by leveraging Feature Store.
A feature store is a tool for storing commonly used features. As data scientists create new features then can log these into feature stores such as Feast and Databricks Feature Store, they can reuse these features across teams and projects. This will benefit teams in multiple ways by reducing compute times for both training and inference, provide consistency in common features and reducing effort for create complex logic.
Read on for information about all six phases.
Comments closedKoen Verbeeck wants to deploy a Powershell-based Azure Function:
In the blog post Azure Function with PowerShell and the Power BI REST API I explained how you could create an Azure Function using the PowerShell scripting language. This Function connected with the Power BI REST API and retrieved the last refresh status of a dataset. Developing the Function is one thing, deploying it is another. In this blog post I’ll guide you through the set-up of a build and release pipeline in Azure Devops. As a prerequisite, the Azure Function and its dependencies (for example the requirements.psd1 file) are all checked into a Git repo. As a reminder, the folder structure looks like this:
Read on for the walkthrough.
Comments closedDrew Skwiers-Koballa announces a new way of handling database projects:
Declarative development creates an environment where developers can focus on creating database objects while relying on the support of tooling locally and and in deployment pipelines to manage applying the differential changes calculated on the current state of the target database. Developers create objects such as tables or stored procedures by writing their definition with CREATE statements in scripts that live in source control just as if it is source code for any component of an application. Existing functionality for SQL projects in Visual Studio, Azure Data Studio, and VS Code provides developers with declarative development capabilities, however the existing SQL project file format has a few limitations. With Microsoft.Build.Sql and SDK-style SQL projects, we look forward to unlocking new scenarios for your development practices.
It does sound interesting.
Comments closedIn this post I want to cover how you can keep your Azure Synapse secrets secret in Azure DevOps. Because you need to do this if you are working with production deployments.
With this in mind, I want to raise more awareness about it and make sure others avoid putting secrets directly in their pipelines like in the below example.
Read on to understand what options are available to you. My preference involves Key Vault references but there are alternatives available.
Comments closedKoen Verbeeck makes use of system databases in a database project:
I have a database project in Visual Studio. Inside the database, I use a couple of system views to fetch some metadata about tables. To make the project build successfully, you need to add a reference to the master database in the project.
That all works fine but there’s a bit more you need to do before Azure DevOps can work with the file. Read on to learn what that thing is.
Comments closedMeagan Longoria solves a problem:
We chose release pipelines over the YAML pipelines because it was easier to manage for the client and pretty quick to set up. While I had done this before, I had a couple of new challenges:
– I was deploying to an Azure SQL managed instance that had no public endpoint.
– There were multiple databases for which there may or may not be a change script to execute in each release.
This took a bit longer than I expected, and I enlisted my coworker Bill to help me work through the final details.
Read on to see how Meagan and Bill solved the problem.
Comments closed