Anubhav Tarar shows us ways of optimizing YARN to run Apache Spark jobs:
1. yarn-client mode: In
clientmode, the driver runs in the client process, and the application master is only used for requesting resources from YARN. To manage the memory first make sure that you have your yarn-site.xml in spark,
- spark.yarn.am.memory: To increase the memory you should set spark.yarn.am.memory property in spark-defaults.conf but make sure that you do not allocate more memory than capacity of node manager which is defined in yarn-site.xml as yarn.nodemanager.resource.memory-mb or you can also give it when you are running spark submit with –conf parameter
For example $SPARK_HOME/bin/spark-submit –conf spark.yarn.am.memory=1024m
Check it out for a few other configuration settings you can tweak.