WARN cluster.YarnScheduler: Initial job has not accepted any resources

apache-spark pyspark yarn taskscheduler
Santanu Dey picture Santanu Dey · Jun 16, 2017 · Viewed 11.9k times · Source

Any spark jobs that I run will fail with the following error message

17/06/16 11:10:43 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Spark version is 1.6, running on Yarn.

I am issuing jobs from pyspark.

And you can notice from the job timeline that it runs indefinitely and no resources are added or removed.1

Answer

Santanu Dey picture Santanu Dey · Jun 16, 2017

First point is that if there are enough resources such as nodes, CPUs and memory available to yarn it can use dynamic allocation to create spark workers with appropriate default cores and memory allocated.

In my case I needed to turn off dynamic allocation as my resource levels were very low.

So from pyspark I set the following values :

conf = (SparkConf().setAppName("simple")
        .set("spark.shuffle.service.enabled", "false")
        .set("spark.dynamicAllocation.enabled", "false")
        .set("spark.cores.max", "1")
        .set("spark.executor.instances","2")
        .set("spark.executor.memory","200m")
        .set("spark.executor.cores","1")

Note: basically the values set here should be less than the actual resources available. However too small values here can lead to out of memory issues, or slow performance issues when your job runs.

The complete code gist of a sample job is available here

Another important point to note for this pyspark case is that Spark on Yarn can run on two modes

  1. cluster mode - the spark driver is run in the spark master node
  2. client mode - the spark driver is run from the client side where the interactive shell is run.

Cluster mode is not well suited to using Spark interactively. Spark applications that require user input, such as spark-shell and pyspark, require the Spark driver to run inside the client process that initiates the Spark application.

Client mode can be set in environment as below
export PYSPARK_SUBMIT_ARGS='--master yarn --deploy-mode client pyspark-shell'

Related questions

Spark Kill Running Application

I have a running Spark application where it occupies all the cores where my other applications won't be allocated any resource. I did some quick research and people suggested using YARN kill or /bin/spark-class to kill the command. However, …

Read More
Do exit codes and exit statuses mean anything in spark?

I see exit codes and exit statuses all the time when running spark on yarn: Here are a few: CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM ...failed 2 times due to AM Container for application_1431523563856_0001_000002 exited with exitCode: 10... ...Exit status: 143. Diagnostics: Container killed on …

Read More
Exception: java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. in spark

I am new apache-spark. I have tested some application in spark standalone mode.but I want to run application yarn mode.I am running apache-spark 2.1.0 in windows.Here is My code c:\spark>spark-submit2 --master yarn --deploy-mode client --executor-cores 4 …

Read More