Brad Llewellyn walks us through the three key data structures in Apache Spark:
We see that creating an RDD can be done with one easy function. In this snippet, sc represents the default SparkContext. This is extremely important, but is better left for a later post. SparkContext offers the .textFile() function which creates an RDD from a text file, parsing each line into it’s own element in the RDD. These lines happen to represent CSV records. However, there are many other common examples that use lines of free text.
It should also be noted that we can use the .collect() and .take() functions to view the contents of an RDD. The difference between .collect() and .take() is that .take() allows us to specify the number of elements we want to retrieve, whereas .collect() returns the entire RDD.
My tendencies are probably skewed pretty heavily, but I live in DataFrames and almost never use raw RDDs anymore.