A Fine Slice Of SQL Server
Tomaz Kastrun is putting together an Advent of Azure Databricks:
Yesterday we started working towards data import and how to use drop zone to import data to DBFS. We have also created our first Notebook and this is where I would like to start today. With a light introduction to notebooks.
Read on for a depiction of notebooks, as well as an example which loads data into the Databricks File System (DBFS).
Comments closedWolfgang Strasser wants to try out Azure Purview:
Basics – Resource group, purview account name (this cannot be changed afterwards) and the location.
As of today (2020-12-06), there are only 5 Azure regions you can choose from to store the Purview metadata. But – in-region scanning from 16 other Azure regions is available in the preview (source)
This is part one of a multi-part series, so stay tuned for more.
Comments closedWolfgang Strasser gives us a once-over on a new service:
Today, at the Azure Data and Analytics event, a new Azure data governance service called Azure Purview (https://aka.ms/AzurePurview) was presented and made available in a public preview.
I have not had a chance to try the actual service, but I found a very interesting video (Microsoft mechanics video) where I took the following screenshots from.
Read on for Wolfgang’s thoughts. It’s definitely a step up from Azure Data Catalog.
Comments closedPaul Andrew shows what happens you you break the ADF concurrency barrier:
Firstly, understanding how these limits apply to your Data Factory pipelines takes a little bit of thinking about considering you need to understand the difference between an internal and external activity. Then you need to think about this with the caveats of being per subscription and importantly per Azure Integration Runtime region.
Assuming you know that, and you’ve hit these limits!
Click through to see what happens. It’s not pretty.
Comments closedCraig Porteous walks us through some of the planning necessary for self-hosted integration runtimes:
If your Data Factory contains a self-hosted Integration runtime, you will need to do some planning work before everything will work nicely with CI/CD pipelines. Unlike all other resources in your Data Factory, runtimes won’t deploy cleanly between environments, primarily as you connect the installed runtime directly to a single Data Factory. (We can add more runtime nodes to a single Data Factory but we cannot share a single node between many data factories*). An excerpt from Microsoft’s docs on Continuous integration and delivery in Azure Data Factory mentions this caveat.
Read on for the consequences and two options available to you.
Comments closedJess Pomfret takes us through running dbachecks on an Azure SQL Database:
Last week I gave a presentation at Data South West on dbachecks and dbatools. One of the questions I got was whether you could run dbachecks against Azure SQL Databases, to which I had no idea. I always try to be prepared for potential questions that might come up, but I had only been thinking about on-premises environments and hadn’t even considered the cloud. The benefit is this gives me a great topic for a blog post.
Click through for the answer.
Comments closedJulie Koesmarno shows off the Kusto Query Language magic in Azure Data Studio notebooks:
To do this, you’ll need to ensure that you have Kqlmagic installed. See Install and set up Kqlmagic in a notebook. Then in a notebook, you can load Kqlmagic with
%reload_ext Kqlmagicin a code cell.The next step is then in a new code cell, you can start connecting to a Log Analytics workspace. There are three ways to do so (roughly – as I’m also learning in this space too):
1. Using Azure Active Directory Device Login authentication.
2. Using Az CLI login
3. Using Client Secret
Read on for one example using Azure AD authentication.
Comments closedTino Zishiri takes us through the concept of the Integration Runtime:
An Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide data integration capabilities such as Data Flows and Data Movement. It has access to resources in either public networks, or hybrid scenarios (public and private networks).
Read on to learn more about what they do and the variety of Integration Runtimes available to you.
Comments closedHong Ooi announces a new package on CRAN:
I’m pleased to announce that the AzureTableStor package, providing a simple yet powerful interface to the Azure table storage service, is now on CRAN. This is something that many people have requested since the initial release of the AzureR packages nearly two years ago.
Azure table storage is a service that stores structured NoSQL data in the cloud, providing a key/attribute store with a schemaless design. Because table storage is schemaless, it’s easy to adapt your data as the needs of your application evolve. Access to table storage data is fast and cost-effective for many types of applications, and is typically lower in cost than traditional SQL for similar volumes of data.
If that sounds like a fit for you, check out the package.
Comments closedHasan Savran wants information:
Indexes let your queries run faster. When you need to adjust your indexing policies, database engines re-indexes your data respecting to your changes. In Cosmos DB, when you change your indexing policies, database engine truncates all your indexes and starts to reindex all your indexes from scratch. You do not want to change your indexing policies when your application is busy. Because your queries can not use the dropped indexes, queries will take longer, and they will cost more Request Units. Also, your queries might not return all the data they supposed to. You can read me my older post about indexes in Cosmos DB.
You may want to monitor re-indexing progress; you may want to disable your application until indexing is completed or warn your team about the re-indexing progress. You can check the re-indexing progress only from SDK, that means you need to write your own code to accomplish this. I have the following code which checks the progress every second. If progress is at %100 then it quits, otherwise it continues to check progress every second until it receives 100 as result.
Hasan has provided us with a script, so check that out.
Comments closed