Praveen Sripati has a two-part series on getting aggregates by year in various Spark languages. In part one, he looks at Python:
Hadoop – The Definitive Guide revolves around the example of finding the maximum temperature for a particular year from the weather data set. The code for the same is here and the data here. Below is the Spark code implemented in Python for the same.
In part 2, he looks at Spark SQL:
In the previous blog, we looked at how find out the maximum temperature of each year from the weather dataset. Below is the code for the same using Spark SQL which is a layer on top of Spark. SQL on Spark was supported using Shark which is being replaced by Spark SQL.Here is a nice blog from DataBricks on the future of SQL on Spark.
There’s no Scala example here, but it’s pretty straightforward as well.