How to limit the number of retries on Spark job failure?

apache-spark yarn
jk-kim picture jk-kim · Aug 2, 2016 · Viewed 32.6k times · Source

We are running a Spark job via spark-submit, and I can see that the job will be re-submitted in the case of failure.

How can I stop it from having attempt #2 in case of yarn container failure or whatever the exception be?

enter image description here

This happened due to lack of memory and "GC overhead limit exceeded" issue.

Answer

Jacek Laskowski picture Jacek Laskowski · Aug 2, 2016

There are two settings that control the number of retries (i.e. the maximum number of ApplicationMaster registration attempts with YARN is considered failed and hence the entire Spark application):

  • spark.yarn.maxAppAttempts - Spark's own setting. See MAX_APP_ATTEMPTS:

      private[spark] val MAX_APP_ATTEMPTS = ConfigBuilder("spark.yarn.maxAppAttempts")
    .doc("Maximum number of AM attempts before failing the app.")
    .intConf
    .createOptional

  • yarn.resourcemanager.am.max-attempts - YARN's own setting with default being 2.

(As you can see in YarnRMClient.getMaxRegAttempts) the actual number is the minimum of the configuration settings of YARN and Spark with YARN's being the last resort.

Related questions

Spark Kill Running Application

I have a running Spark application where it occupies all the cores where my other applications won't be allocated any resource. I did some quick research and people suggested using YARN kill or /bin/spark-class to kill the command. However, …

Read More
Apache Spark: The number of cores vs. the number of executors

I'm trying to understand the relationship of the number of cores and the number of executors when running a Spark job on YARN. The test environment is as follows: Number of data nodes: 3 Data node machine spec: CPU: Core i7…

Read More
Where are logs in Spark on YARN?

I'm new to spark. Now I can run spark 0.9.1 on yarn (2.0.0-cdh4.2.1). But there is no log after execution. The following command is used to run a spark example. But logs are not found in the history server as in …

Read More