Guy Shilo explains why we should care about Apache Livy:
Apache Livy is an open source server that exposes Spark as a service. Its backend connects to a Spark cluster while the frontend enables REST API. This enables running it as the organization’s Spark gateway and even run in in docker containers.
Not only it enables running Spark jobs from anywhere, but it also enables shared Spark context and a shared RDD cache among all it’s users which is time and memory saving.
I will demonstrate here how to setup Apache Livy on one of the cluster’s nodes and on a separate server.
Click through for the demonstration.