Tristan Robinson shows us how to run Spark Streaming within Azure Databricks:
Real-time stream processing is becoming more prevalent on modern day data platforms, and with a myriad of processing technologies out there, where do you begin? Stream processing involves the consumption of messages from either queue/files, doing some processing in the middle (querying, filtering, aggregation) and then forwarding the result to a sink – all with a minimal latency. This is in direct contrast to batch processing which usually occurs on an hourly or daily basis. Often is this the case, both of these will need to be combined to create a new data set.
In terms of options for real-time stream processing on Azure you have the following:
-
Azure Stream Analytics
-
Spark Streaming / Storm on HDInsight
-
Spark Streaming on Databricks
-
Azure Functions
Click through for more.