Kara Annanie has five things you should know about the Databricks Unified Analytics Platform:
4. A Spark Dataframe is not the same as a Pandas/R Dataframe
Spark Dataframes are specifically designed to use distributed memory to perform operations across a cluster whereas Pandas/R Dataframes can only run on one computer. This means that you need to use a Spark Dataframe to realize the benefits of the cluster when coding in Python or R within Databricks.
This is a nice set of tips for people getting started with Spark and Databricks.