Structured Streaming In Apache Spark: A new high-level API for streaming
Databricks’ engineers and Apache Spark committers Matei Zaharia, Tathagata Das, Michael Armbrust and Reynold Xin expound on why streaming applications are difficult to write, and how Structured Streaming addresses all the underlying complexities.
There’s quite a bit of reading material on the other side.