Press "Enter" to skip to content

Category: Source Control

Azure Data Factory and Source Control

Published 2021-01-12 by Kevin Feasel

Ahmad Yaseen shows how you can save Azure Data Factory pipelines in source control:

To overcome these limitations, Azure Data Factory provides us with the ability to integrate with a GIT repository, such as Azure DevOps or GitHub repository, that helps in tracking and versioning the pipelines changes, and incrementally save the pipeline changes during the development stage, without the need to validate the incomplete pipeline, preventing these changes from being lost in case of any crash or failure. In this case, you will be able to test the pipeline, revert any change that is detected as a bug, and publish the pipeline to the Data Factory when everything is developed and validated successfully.

Click through for the setup instructions.

Comments closed

Adding a Database Project to GitHub

Published 2020-12-30 by Kevin Feasel

Elizabeth Noble shows how you can get your brand new Azure Data Studio project into GitHub:

Once you have the database project created, you’ll want to get your database project added to source control so that you (and others) can modify and manage your database code. This next step is the beginning of allowing you and others to work on the same databases and minimize the risk of overwriting someone else’s work or deploying the wrong code to Production.

Tools like GitHub Desktop and SourceTree have definitely made things easier, especially for the happy path scenarios.

Comments closed

Thoughts on Using Source Control

Published 2020-09-23 by Kevin Feasel

Kevin Chant shares some thoughts:

In this post I want to cover more thoughts about SQL Server professionals using version control. Because I have had some interesting conversations since my last post about it.

In a previous post I covered how SQL Server professionals can benefit from using version control. Which you can read in detail here.

Now I want to clarify a few things relating to it as well.

Read on for those thoughts.

Comments closed

Tying Azure Data Factory to Source Control

Published 2020-04-06 by Kevin Feasel

Eddy Djaja explains why you really want to tie Azure Data Factory to your source control:

Azure Data Factory (ADF) is Microsoft’s ETL or more precise: ELT tool in the cloud. For more information of ADF, Microsoft puts the introduction of ADF in this link: https://docs.microsoft.com/en-us/azure/data-factory/introduction. As some have argued if ADF will replace or complement the “on-premise”  SSIS, it is uncertain and only time can tell what will happen in the future.
Unlike SSIS, the authoring of ADF does not use Visual Studio. ADF authoring uses a web browser to create ADF components, such as pipelines, activities, datasets, etc. The simplicity of authoring ADF may confuse the novice developers on how ADF components are saved, stored and published. When logging to ADF for the first time after creating an ADF, the authoring is in the ADF mode. How do we know?

Click through for the explanation and some resources on how to do it.

Comments closed

A Script for Peer Reviews of Code

Published 2020-01-30 by Kevin Feasel

Paul Andrew shares with us a script which is useful for peer reviewing code before check-in:

Does the code include good comments? Things that explain the reason why logic has been implemented to assist future developers looking at the same code.

All to often I see code comments written that just translate from code to English and tell me what is happening. What should be fairly obvious to the reviewer as they read the code. Why is so much more important.

It’s a 20-point checklist, but worth reviewing and adapting for your own purposes.

Comments closed

Distributing Notebooks

Published 2019-12-12 by Kevin Feasel

Grant Fritchey wants to know where to buy notebooks and notebook accessories:

I’m myopically focused at the moment on Azure Data Studio, but there are a lot of other places and ways to create or consume notebooks. However, I’m going to keep my focus.

The issue I’m running into, is distributing the notebooks.

There are a lot of great comments. Before reading them, here’s my answer:

  • GitHub repos, like Grant mentions. They’re good, though I have the same feeling about a production notebook that I do about an SSIS package: notebooks are binaries (after a fashion). For pedagogical purposes, I’ll absolutely slap notebooks into GitHub, typically without data. But for a real data science project, those notebooks can get hefty when you store all of the data in them, and it’s really hard to diff the JSON to understand what changed.
  • Binder and Azure Notebooks are services which let you host notebooks remotely. Binder reads from a GitHub repo and spins up a virtual environment for you. Azure Notebooks lets you run notebooks (including F# notebooks) against free VMs in Azure, or you can use your own VM for more power. Azure Notebooks let you fork projects pretty easily. I haven’t used Google Colab but it looks pretty similar to Azure Notebooks.
  • When you start up Jupyter Notebooks, you’re really starting a server. You can have a server running in your environment with your team’s notebooks. I’d probably still drop them in source control as well.
Comments closed

Forking GitHub Repos and Contributing to Open Source Projects

Published 2019-12-03 by Kevin Feasel

Rob Sewell takes us through the process of contributing to an open source project:

– Fork the repository into your own GitHub

– Clone the repository to your local machine

– Create a new branch for your changes

– Make some changes and commit them with useful messages

– Push the changes to your repository

– Create a Pull Request from your repository back to the original one

You will need to have git.exe available which you can download and install from https://git-scm.com/downloads if required

For bonus points, we learn that Shane O’Neill doesn’t use the Oxford comma.

Comments closed

A Git Cheat Sheet

Published 2019-12-03 by Kevin Feasel

Kendra Little has a cheat sheet for working with Git:

I created a cheat sheet for the Git Command Line Interface to go along with my Git tutorial for SQL Change Automation video. I find the Git CLI to be very friendly and easier to learn than a GUI interface.

Given the number of “How do I extricate myself from this Git mess?” messages in my company chat, I’m not sure I’d call the Git CLI friendly. Nonetheless, Kendra does a great job of putting together most of the common commands in an easy guide.

Comments closed

SQL Server Trends Worth Watching

Published 2019-11-13 by Kevin Feasel

Grant Fritchey follows up on a Kevin Hill tweet:

There are a million things to learn about in our rapidly shifting technological landscape, but I think this assessment, especially the way it was put, “no longer justify ignoring” really nails some of the fundamentals.

Let’s talk about why you can no longer ignore Docker, Git and DBATools either.

If you’re a DBA and aren’t familiar with Docker, Git, or DBATools, that’s a pretty good trio of things to spend some time learning. You can survive without them, but you’re more likely to thrive if you know them.

Comments closed