Press "Enter" to skip to content

Day: December 6, 2019

Querying Pulsar Streams with Apache Flink

Published 2019-12-06 by Kevin Feasel

Sijie Guo and Markos Sfikas show how we can interact with Apache Pulsar using Apache Flink:

The latest integration between Flink 1.9.0 and Pulsar addresses most of the previously mentioned shortcomings. The contribution of Alibaba’s Blink to the Flink repository adds many enhancements and new features to the processing framework that make the integration with Pulsar significantly more powerful and impactful. Flink 1.9.0 brings Pulsar schema integration into the picture, makes the Table API a first-class citizen and provides an exactly-once streaming source and at-least-once streaming sink with Pulsar. Lastly, with schema integration, Pulsar can now be registered as a Flink catalog, making running Flink queries on top of Pulsar streams a matter of a few commands. In the following sections, we will take a closer look at the new integrations and provide examples of how to query Pulsar streams using Flink SQL.

Read on to see this integration in action.

Comments closed

Counting Tidyverse Package Arguments

Published 2019-12-06 by Kevin Feasel

Theo Roe has fun figuring out which tidyverse packages have the greatest number of available arguments in functions:

Before we start anything, I’d like to mention that most of the hard work came from nsaunders and his great blog post Idle thoughts lead to R internals: how to count function arguments.

Let’s get started.

The aim of this blog is to capture the number of arguments present in each function with packages of the tidyverse. 

Click through to see the code, as well as some methods of visualizing the results (methods which you can use in other situations).

Comments closed

Handling Forbidden XML Characters with SQL Server

Published 2019-12-06 by Kevin Feasel

Slava Murygin shows how we can use Unicode characters to make XML appear to display special characters:

That is very known issue that SQL Server’s XML does not accept characters “&”, “<” and “>”.
There are two more forbidden XML characters ” ‘ ” and ” ” ” (single and double quotes), but SQL Server mostly accept them.

The common solution is to replace these characters by their codes.
Would say we have a silly sentence: “Anne & Robin collect > “berries” than Jane & Kevin, but < than Ivan & Lucy.”

Slava’s post is specifically geared toward wanting to view the characters as-is, not store them for later display. I’m not sure how often that comes up, but it’s a valid use case.

Comments closed

Learning to Learn

Published 2019-12-06 by Kevin Feasel

Buck Woody has a great post on learning how to learn:

In this new world of fast-paced learning, you’ll often find that you have to “throw away” what you’ve learned, meaning that a new language or tool is out now that requires your attention, and you won’t return to the one you know now. That doesn’t mean your hard study was wasted, because you’ll often find that new technology builds on the one you just learned, but I find that Type-A technologists are loath to drop something they just learned. You’ll have to get over that – it’s the way it is.

However, it can be true that once you learn something, it may be in an area that you just had to come up to speed on quickly, or it has “staying power” and will be around for a while. In that case, take this same process, and repeat all the steps, taking time to fill in the gaps and go much deeper in the areas you didn’t spend time on during your speed learning.

I really liked this post. The first thing it reminded me of was Sir Francis Bacon’s Of Studies (pdf, but with bonus content from Samuel Johnson), specifically the part about how we should superficially breeze through some books, but that others we must digest. The same goes with technologies.

Comments closed

Conversion Failed when Converting ‘NULL’ to Int

Published 2019-12-06 by Kevin Feasel

Kenneth Fisher has a fun error for us:

I love this error. Primarily because it demonstrates two very important things.

1. Errors matter. Make sure when you ask someone for help you give them the exact error and circumstances causing the error.
2. Experience matters. If you’ve been working with database development for a while you can probably pinpoint exactly what’s causing this error just from the error.

When I was handed this error from one of my co-workers I started by telling them exactly what was wrong, and then out of curiosity started a quick poll.

The answer is about as straightforward as it gets, and yet a pretty good percentage of people won’t get it on the first try.

Comments closed

Custom Power BI Visuals with Charticulator

Published 2019-12-06 by Kevin Feasel

Kasper de Jonge shows how we can use Charticulator to build out custom visuals in Power BI:

Recently I attended an excellent session by Andy Kirk on the state of data visualization for 2019. One of the tools Andy is most excited about is called Charticulator. For those of you that haven’t heard about it, Charticulator is an open source project from Microsoft Research. Using a web UI you can design almost any charts by interactively specifying constraints (NO code).

The best news is that you can use these visuals directly in Power BI. You can even use your own data coming from the PBI data model. This came as news for many at Andy Kirk’s session so decided to do a quick post on it. Building very custom charts entices a lot of data artists but it is often hard to bring this to main stream BI product. The ease of the Charticulator and Power BI integration brings this type of data visualizations to a much broader audience.

This is a lot easier than telling people to learn D3, though Charticulator will necessarily have more limitations than writing all of the code yourself.

Comments closed

Calculating Last Year to Date with DAX

Published 2019-12-06 by Kevin Feasel

Reza Rad wants to compare prior years to the current year, using year-to-date comparisons:

I have previously written about how to calculate the same period last year calculation and compare this year’s values with the last year’s values. However, sometimes, you don’t yet have the full year, especially for the current year. You might want to compare this year’s value with the last year’s value up until the same day but last year. This is what I call same period last year to date. Here in this blog article, I’ll explain how you can do that using DAX in Power BI. To learn more about Power BI, read Power BI from Rookie to Rock Star.

Click through to see how it’s done.

Comments closed

Azure Data Factory Pipelines

Published 2019-12-06 by Kevin Feasel

Cathrine Wilhelmsen continues a series on Azure Data Factory with a discussion of pipelines:

Pipelines are sorted by name, so I recommend that you decide on a naming convention early in your project. And yeah, I keep saying this to everyone else, but then I can never decide on how to name my own pipelines, haha 🙂 Don’t worry if you end up renaming your pipelines several times while you work on your project. It happens, and that’s completely fine, but try to stick to some kind of naming convention throughout your project.

In addition to naming conventions, you can create folders to organize your pipelines. Click the actions ellipsis next to the pipelines group, then click new folder.

Read on for more.

Comments closed