A Fine Slice Of SQL Server
Chris Taylor has a point updates to jump in Azure Kubernetes Service:
As it is late at night my brain wasn’t working as it should be but thought I’d put a quick blog out there to say that if you are on v1.11.5 and want to upgrade to >= v1.13.10 then you have to do this in a 2 stage process by upgrading to v1.12.8 first:
Fortunately, upgrading is pretty easy using the Azure command line or even the Azure portal.
Comments closedAzure Data Factory’s Mapping Data Flows have built-in capabilities to handle complex ETL scenarios that include the ability to handle flexible schemas and changing source data. We call this capability “schema drift“.
When you build transformations that need to handle changing source schemas, your logic becomes tricky. In ADF, you can either build data flows that always look for patterns in the source and utilize generic transformation functions, or you can add a Derived Column that defines your flow’s canonical model.
Click through for the discussion and comparison. Schema drift has been the bane of Integration Services’s existence, so it’s good to see them tackling the idea in Azure Data Factory.
Comments closedThe SQL Engine we are running in Azure SQL Database is the very latest version of the same engine customers run on their own servers, except we manage and update it. To update SQL Server or the underlying infrastructure (i.e. Service Fabric or the operating system), we must stop the SQL Server process. If that process hosts the primary database replica, we move the replica to another machine (requiring a failover).
During failover, the database may be offline for a second and still meet our 99.995% SLA. However, failover of the primary replica impacts workload because it aborts in-flight queries and transactions. We built features such as resumable index (re)build and accelerated database recovery to address these situations, but not all running operations are automatically resumable. It may be expensive to restart complex queries or transactions that were aborted due to an upgrade. So even though failovers are quick, we want to avoid them.
Read on to see how they do it. There’s no on-prem analogue yet, though perhaps that will come in time.
Comments closedChris Adkin shows off parallelism in Azure DevOps continuous integration pipelines:
A SQL Server data tools project is checked out of GitHub, built into a DacPac, four containerized SQL Server instances are spun up using clones of the ‘Seed’ docker volume. The DacPac is applied to a database running inside each container, which a tSQLt test is then executed against, finally, at the end very end the tSQLt results are aggregate and published.
This is an interesting approach to the problem of lengthy tests: run them on several separate machines concurrently.
Comments closedSamir Behara takes us through troubleshooting AWS Database Migration Service issues:
For troubleshooting any issues with AWS DMS, it is necessary to have logs enabled. The DMS logs would typically give a better picture and helps find errors or warnings that would indicate the root cause of the failure. If the logs are not available there is nothing much you can do from a detailed troubleshooting analysis perspective. So basically next step is to turn on DMS logs and kick the job again and validate if the errors are captured in the logs.
If logs are not enabled, you need to set up a new task with logging enabled so if and when it errors out, you can take a look and troubleshoot the same.
I’ll save my full rant for another day, but I’m not that impressed with DMS. It could be a failing on my part, though.
Comments closedBub Pusateri has a helpful script for us:
Last year I wrote about how to upload data to Azure Blob Storage Archive Tier, and included a PowerShell script to do so. It’s something I use regularly, as I have hundreds of gigabytes of photos and videos safely (and cheaply!) stored in Azure Blob Storage using Archive Tier.
But read on for a recent announcement to make the process easier.
Comments closedArun Sirpal explains what an Azure Shared Access Signature is:
Using a Shared Access Signature (SAS) is usually the best way to control access rights to Azure storage resources (like a container for backups) without exposing the primary / secondary storage keys. It is based on a URI and this is what I want to look at today.
Read on to learn about the components which make up a Shared Access Signature.
Comments closedJohn Sterrett has run into a few issues with Azure Backup for SQL Server VMs:
Having the ability to backup new databases automatically is taken for granted. So much, that I noticed that Azure Backup for SQL Server VM’s will not automatically backup new databases for you. That’s right. Make sure you remember to go in and detect and select your new database every time you add a database or you will not be able to recover.
Azure Backup for SQL Server VM’s has an interesting feature called Autoprotect. This should automatically backup all your databases for you. Unfortunately, this does not work. Yes, I double-checked by enabling autoprotect for a VM and I added a new database. The database didn’t get backed up so I had to manually add the database.
Some of these seems like it’s easy enough to fix, so hopefully the product gets better over time.
Comments closedMapping Data Flows
One of the many data flows from Microsoft these days providing, for the first time, data transformation capabilities within Data Factory. This is not a U-SQL script or Databricks notebook that is orchestrated from Data Factory, but a tool integrated. This means that you can reuse (many of) the datasets you have defined in Data Factory, while in Databricks you don’t.Mapping Data Flows runs on top of Databricks, but the cluster is handled for you and you don’t have to write any of that Scala code yourself.
Read on for the full comparison.
Comments closedJamie Wick helps us figure out how to keep our data local while using Azure services:
One of the challenges many organizations face when beginning to work with Azure applications (PowerBI, PowerApps, Flow, etc.) is that their data is on-premise and the applications are hosted in the cloud. Moving the data to the cloud is often cost-prohibitive and there can be operational requirements that prevent the data, or the systems hosting it, from being relocated to the cloud.
So, how can on-prem data be used with Azure apps?
Read on for more.
Comments closed