How to set amount of Spark executors?

java scala cluster-computing apache-spark yarn
Roman Nikitchenko picture Roman Nikitchenko · Oct 2, 2014 · Viewed 51.1k times · Source

How could I configure from Java (or Scala) code amount of executors having SparkConfig and SparkContext? I see constantly 2 executors. Looks like spark.default.parallelism does not work and is about something different.

I just need to set amount of executors to be equal to cluster size but there are always only 2 of them. I know my cluster size. I run on YARN if this matters.

Answer

A. One picture A. One · Jul 20, 2016

You could also do it programmatically by setting the parameters "spark.executor.instances" and "spark.executor.cores" on the SparkConf object.

Example:

SparkConf conf = new SparkConf()
      // 4 executor per instance of each worker 
      .set("spark.executor.instances", "4")
      // 5 cores on each executor
      .set("spark.executor.cores", "5");

The second parameter is only for YARN and standalone mode. It allows an application to run multiple executors on the same worker, provided that there are enough cores on that worker.

Related questions

How to use java.String.format in Scala?

I am trying to use a .format method of a string. But if I place %1, %2, etc. in the string, java.util.UnknownFormatConversionException is thrown pointing to a confusing Java source code piece: private void checkText(String s) { int idx; // If …

Read More
What are the best use cases for Akka framework

I have heard lots of raving about Akka framework (Java/Scala service platform), but so far have not seen many actual examples of use cases it would be good for. So I would be interested in hearing about things developers …

Read More
Add jars to a Spark Job - spark-submit

True ... it has been discussed quite a lot. However there is a lot of ambiguity and some of the answers provided ... including duplicating jar references in the jars/executor/driver configuration or options. The ambiguous and/or omitted details Following …

Read More