List of spark-submit options

Question

There are a ton of tunable settings mentioned on Spark configurations page. However as told here, the SparkSubmitOptionParser attribute-name for a Spark property can be different from that property's-name.

For instance, spark.executor.cores is passed as --executor-cores in spark-submit.

---

Where can I find an exhaustive list of all tuning parameters of Spark (along-with their SparkSubmitOptionParser property name) that can be passed with spark-submit command?

Answer 1

While @suj1th's valuable inputs did solve my problem, I'm answering my own question to directly address my query.

---

• You need not look up for SparkSubmitOptionParser's attribute-name for a given Spark property (configuration setting). Both will do just fine. However, do note that there's a subtle difference between there usage as shown below:

  spark-submit --executor-cores 2

  spark-submit --conf spark.executor.cores=2

  Both commands shown above will have same effect. The second method takes configurations in the format --conf <key>=<value>.

• Enclosing values in quotes (correct me if this is incorrect / incomplete)

  (i) Values need not be enclosed in quotes (single '' or double "") of any kind (you still can if you want).

  (ii) If the value has a space character, enclose the entire thing in double quotes "" like "<key>=<value>" as shown here.

• For a comprehensive list of all configurations that can be passed with spark-submit, just run spark-submit --help

• In this link provided by @suj1th, they say that:

  configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file.

  If you are ever unclear where configuration options are coming from, you can print out fine-grained debugging information by running spark-submit with the --verbose option.

---

Following two links from Spark docs list a lot of configurations:

• Spark Configuration
• Running Spark on YARN