Flink And Kafka Streams

Neha Narkhede and Stephan Ewen compare Apache Flink versus Kafka Streams:

Before Flink, users of stream processing frameworks had to make hard choices and trade off either latency, throughput, or result accuracy. Flink was the first open source framework (and still the only one), that has been demonstrated to deliver (1) throughput in the order oftens of millions of events per second in moderate clusters, (2) sub-second latency that can be as low as few 10s of milliseconds, (3) guaranteed exactly once semantics for application state, as well as exactly once end-to-end delivery with supported sources and sinks (e.g., pipelines from Kafka to Flink to HDFS or Cassandra), and (4) accurate results in the presence of out of order data arrival through its support for event time. Flink is based on a cluster architecture with master and worker nodes. Flink clusters are highly available, and can be deployed standalone or with resource managers such as YARN and Mesos. This architecture is what allows Flink to use a lightweight checkpointing mechanism to guarantee exactly-once results in the case of failures, as well allow easy and correct re-processing via savepoints without sacrificing latency or throughput. Finally, Flink is also a full-fledged batch processing framework, and, in addition to its DataStream and DataSet APIs (for stream and batch processing respectively), offers a variety of higher-level APIs and libraries, such as CEP (for Complex Event Processing), SQL and Table (for structured streams and tables), FlinkML (for Machine Learning), and Gelly (for graph processing). Flink has been proven to run very robustly in production at very large scale by several companies, powering applications that are used every day by end customers.

The upshot is that the two products don’t do exactly the same thing, and there might be room in your organization for the two of them.

The Joy Of Hyperparameters

Koos van Strien shows how to tune hyperparameters using Azure ML:

Today, we’ll focus on tuning the model’s properties. We won’t discuss the details of all properties (you can easily look that up in the docs), instead we’ll look at how to test for different parameter combinations insize Azure ML Studio.

As soon as you click on an untrained model inside your experiment, you’ll be presented with some parameters – or, in ML parlance, hyperparameters – you can tweak.

Parameter tuning is pretty easy using Azure ML.

Tornado Visual

Devin Knight looks at the Tornado chart:

  • The Tornado has a few limitation that should be aware of before using

    • If there’s a legend value it should only have 2 distinct values

    • Each distinct category values is a separate bar with left or right parts

    • Alternatively, you can have two measure values and compare them without  a legend

I’m split on whether I like the tornado or not.  It is intuitive and information-dense, which are two major factors in its favor.  It is, however, difficult to read and compare.  This seems like a useful “big picture” chart, but you’d want to organize the data in a different way when you start drilling down.

Powershell Workflows

Cody Konior has a beef with Powershell workflows:

That’s inexplicable.

One thing which does make it all work is setting $PSRunInProcessPreference which, “If this variable is specified, all activities in the enclosing scope are run in the workflow process.” Unfortunately that doesn’t explain what’s really going on and what the impacts are, so I won’t use it. But here it is turning the original failing script into a working one.

I’ve never used Powershell workflows.  It sounds like potentially an exasperating experience.

Why Heaps?

Daniel Hutmacher makes an argument for when to use a heap instead of a clustered index:

I’m an outspoken advocate of always using a clustered index on each and every table you create as a matter of best practice. But even I will agree that there’s a case for using the odd heap now and then.

Read on for indicators that you might be better served with a heap.

Renaming SQL Servers

Wayne Sheffield shows what to do when you need to rename your SQL Server instance:

Sometimes you make a mistake, and forget to rename a syspred’d server before installing SQL Server. Or perhaps your corporate naming standard has changed, and you need to rename a server. Maybe you like to waste the time involved in troubleshooting connection issues after a server rename. In any case, you now find yourself where the name of the SQL Server is different than the physical name of the server itself, and you need to rename SQL Server to match the server’s physical name.

You could always rerun the setup program to rename the server. Fortunately, SQL Server provides an easier way to do this. You just need to run two stored procedures:sp_dropserver and sp_addserver.

Click through for details, including important considerations.

CHECKDB And Indexes On Persisted Computed Columns

Kevin Feasel



Arun Sirpal diagnoses a slower-than-usual DBCC CHECKDB run:

All the signs of CHECKDB Latch contention.

DBCC – OBJECT – METADATA this latch can be a major bottleneck for DBCC consistency checks when indexes on computed columns exist.  As a side note DBCC_Multiobject scanner  is used to get the next set of pages to process during a consistency check.

Read on for the details and Arun’s solution.

Hadoop For .NET Developers

Kevin Feasel



Elton Stoneman has a new Pluralsight course out:

My latest Pluralsight course is out now:

Hadoop for .NET Developers

It takes you through running Hadoop on Windows and using .NET to write MapReduce queries – proving that you can do Big Data on the Microsoft stack.

The course has five modules, starting with the architecture of Hadoop and working through a proof-of-concept approach, evaluating different options for running Hadoop and integrating it with .NET.

I’ve liked Elton’s courses, as he’s one of the few trainers who really takes the time to show how you can integrate .NET languages into a Hadoop ecosystem; the general philosophy is “go learn Java and Scala and Python and …”

Self-Paced HDInsight Training

Ashish Thapliyal introduces three EdX courses on HDInsight:

Implementing Real-Time Analysis with Hadoop in Azure HDInsight

Start course

In this four week course, you’ll learn how to implement low-latency and streaming Big Data solutions using Hadoop technologies like HBase, Storm, and Spark on Microsoft Azure HDInsight.

Course Syllabus

Use HBase to implement low-latency NoSQL data stores.
Use Storm to implement real-time streaming analytics solutions.
Use Spark for high-performance interactive data analysis.

These are free courses on EdX.  I personally wouldn’t bother getting the certificate, but hey, it’s your money.

Hortonworks HDP 2.5 Available

Hortonworks has a new version of their data platform, 2.5:

We are very pleased to announce that the Hortonworks Data Platform (HDP) Version 2.5 is now generally available for download. As part of a Open and Connected Data Platforms offering from Hortonworks, HDP 2.5 brings a variety of enhancements across all elements of the platform spanning data science, data access to security to governance.

At Hadoop Summit 2016 San Jose on 06/28/2016, we unveiled the latest innovation package within Hortonworks Data Platform 2.5.

The top points of interest:  Spark 2, Kafka 0.10.0, Ambari 2.4, and Storm 1.0.1.  These are four big projects with major improvements.  Looks like I’ve got something to do this weekend…


June 2018
« May