Push-Based Alerting With Kafka Streams

Robin Moffatt shows how to take syslog data and create a notification app using Python and Kafka Streams:

Now we can query from it and show the aggregate window timestamp alongside the result:

ksql> SELECT ROWTIME, TIMESTAMPTOSTRING(ROWTIME, 'yyyy-MM-dd HH:mm:ss'), \HOST, INVALID_LOGIN_COUNT \FROM INVALID_USERS_LOGINS_PER_HOST;1521644100000 | 2018-03-21 14:55:00 | rpi-03 | 11521646620000 | 2018-03-21 15:37:00 | rpi-03 | 21521649080000 | 2018-03-21 16:18:00 | rpi-03 | 11521649260000 | 2018-03-21 16:21:00 | rpi-03 | 41521649320000 | 2018-03-21 16:22:00 | rpi-03 | 21521649080000 | 2018-03-21 16:38:00 | rpi-03 | 2

In the above query I’m displaying the aggregate window start time, ROWTIME (which is epoch), and converting it also to a display string, using TIMESTAMPTOSTRING. We can use this to easily query the stream for a given window of interest. For example, for the window beginning at 2018-03-21 16:21:00 we can see there were four invalid user login attempts. We can easily check the source data for this, using the ROWTIME in the above output for the window (16:21 – 16:22) as the bounds for the predicate:

It’s a very interesting use case.

Related Posts

How Spark Works: RDDs And DAGs

Shubham Agarwal gets into the way that Spark translates operations on Resilient Distributed Datasets into actions: When we do a transformation on any RDD, it gives us a new RDD. But it does not start the execution of those transformations. The execution is performed only when an action is performed on the new RDD and […]

Read More

Creating Seaborn Plots With R

Abdul Majed Raja shows how to call Python from R and build plots using the Seaborn Python package: The reticulate package provides a comprehensive set of tools for interoperability between Python and R. The package includes facilities for: Calling Python from R in a variety of ways including R Markdown, sourcing Python scripts, importing Python […]

Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

Categories

April 2018
MTWTFSS
« Mar  
 1
2345678
9101112131415
16171819202122
23242526272829
30