Leela Prasad shares a few sample configuration settings for Spark-Submit jobs:
Before going further let’s discuss on the below parameters which I have given for a Job.
spark.executor.cores=5
spark.executor.instances=3
spark.executor.memory=20g
spark.driver.memory=5g
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.maxExecutors=10
Click through to see what these do and why Leela chose these settings. The Spark documentation has the full list of settings but it’s good to hear explanations from practitioners.