Press "Enter" to skip to content

Day: May 24, 2021

Data Pipeline Error Handling with Apache NiFi

Published 2021-05-24 by Kevin Feasel

Pieter Humphrey gives us a few techniques for handling data pipeline errors when running Apache NiFi:

The more complex the model, the more possible sources of problems exist. Forecasting every single potential problem is, of course, impossible. Identifying the most important ones and providing self-solving solutions can greatly reduce the operational uncertainty of our NiFi pipeline and improve its robustness.

To see how to do this analysis, we will consider four possible strategies: one external and three internal. They certainly do not cover all potential error scenarios, they are just examples that we can extrapolate from, and inform how to handle other potential failure domains.

Click through for an overview of the topic as well as those four techniques.

Comments closed

Connectivity Modes for Cosmos DB

Published 2021-05-24 by Kevin Feasel

Hasan Savran takes us through two ways to connect to Cosmos DB:

 There are two ways to connect to the Azure Cosmos DB. You need to specify the way you want to connect to Azure Cosmos DB in SDK. The way that you pick for connectivity mode can make a big difference for your application’s performance. 

     First connectivity mode is Gateway Mode. This is the default way to connect Azure Cosmos DB in earlier version of SDK platforms. This method is mostly for applications that stays behind corporate firewall or It is for environments that have a limited number of socket connections. If your company have strict firewall restrictions, Gateway mode is the way to go for sure. Gateway mode uses standard HTTPS port and single endpoint.

Read on for the second mode, as well as the pros and cons of using each mode.

Comments closed

Azure Purview Support for Azure SQL Database Views

Published 2021-05-24 by Kevin Feasel

Wolfgang Strasser looks at a new feature in Azure Purview:

There was one big thing missing so far – the scanning of SQL Server / Azure SQL Views. Which – well – in many cases (customer databases) was a huge loss of information in the data catalog.

This known limitation is listed on the documentation page but many of us overread this sentence.

But check out Wolfgang’s most recent finding and it’s clear that the team is working on it.

Comments closed

So You Want Database Documentation

Published 2021-05-24 by Kevin Feasel

Joey D’Antoni gives us a primer on database documentation techniques:

Although writing better queries and building the right indexes are important parts of improving database performance, building clear database documentation can also contribute to this goal by helping you understand your database architecture. Painting a clear picture of the structure of your database gives you insight into your data flows and helps you identify redundant data and clarify business processes.

Let’s take a look at a few approaches you can take to documenting your database and your data, depending on the nature of your application.

I do like the idea of using extended properties for documentation purposes, though for the longest time, the tooling to show those extended properties was fairly limited and it was easy to miss them on scripting.

Comments closed

One…Million IO Requests

Published 2021-05-24 by Kevin Feasel

Sean Gallardy wins the jackpot:

If, somehow, you’ve managed to see this error in your errorlog then congratulations, you’ve won an instance of SQL Server that probably won’t be doing much.

I found out about this message a few months ago, but it has been in the product for years and I went this long without ever even knowing it existed (congrats me!) until I was asked about it and coincidentally ended up finding it in an errorlog the same week. Clearly, I have too much fun packed into my weeks. I asked around, only one other person had ever found this in an errorlog before… that’s either impressive, depressing, or some perfect quantity of both – mellow it out to a smooth melancholy.

Click through to see more information about the 1000000 IO error message and when you might find it.

Comments closed

Enforcing Powershell Named Parameters

Published 2021-05-24 by Kevin Feasel

Dale Hirt is the law:

Line 3 is a named parameter called $badParam. This becomes important a little later. Lines 4, 5, 6, and 7 are named parameters. Now, how can we enforce that someone uses those named parameters.

Read on for an interesting technique to ensure that your callers are using named parameter rather than positional parameter calls.

Comments closed