Grabbing Spark With sbt

Kevin Feasel

2017-01-16

Spark

Ian Hellström shows how to create an sbt script to get the a particular version of Spark:

If you have already installed sbt on your machine, read on. If not, have a look here on how to set up your machine.

With sbt available, create a folder in which you can play around, your ‘sandbox’. I’ll assume you have created the folder under /path/to/sandbox. On Windows, also create a sub-folder inside it for Spark’s so-called warehouse directory. Let’s call that sub-folder ‘warehouse’.

Click through for more details.

Related Posts

ACID Transactions on Spark

Achilleus explains one of the big announcements at Spark+AI Summit 2019: Delta Lake is basically a compute layer that would sit on top of your existing On Prem HDFS cluster, your favourite Cloud storage or even run it locally on your laptop(Best part)! Data is stored on the above-mentioned storage as versioned Parquet files. Any data that is read using […]

Read More

Spark+AI Summit 2019 Announcements

Victoria Holt creates a roundup of Spark+AI Summit 2019 announcements: Rohan Kumar  of Microsoft announced .NET for Apache Spark, making Apache Spark accessible to .NET developers – Git Hub I’m very happy that the Spark for .NET team added F# support. Spark is so much nicer when using functional programming.

Read More

Categories

January 2017
MTWTFSS
« Dec Feb »
 1
2345678
9101112131415
16171819202122
23242526272829
3031