How to set Spark application exit status?

apache-spark yarn exit-code spark-submit
roh picture roh · Jan 31, 2017 · Viewed 8k times · Source

I'm writing a spark application and run it using spark-submit shell script (using yarn-cluster/yarn-client)

As I see now, exit code of spark-submit is decided according to the related yarn application - if SUCCEEDED status is 0, otherwise 1.

I want to have the option to return another exit code - for a state that my application succeeded with some errors.

Is it possible? to return different exit code from the application?

I tried to use System.exit() but didn't succeed...

Thanks.

Answer

code picture code · Feb 3, 2017

It is possible in client mode but not in cluster mode. You have a workaround for cluster mode.

My answer to this question should help you.

Related questions

Spark Kill Running Application

I have a running Spark application where it occupies all the cores where my other applications won't be allocated any resource. I did some quick research and people suggested using YARN kill or /bin/spark-class to kill the command. However, …

Read More
Apache Spark: The number of cores vs. the number of executors

I'm trying to understand the relationship of the number of cores and the number of executors when running a Spark job on YARN. The test environment is as follows: Number of data nodes: 3 Data node machine spec: CPU: Core i7…

Read More
Where are logs in Spark on YARN?

I'm new to spark. Now I can run spark 0.9.1 on yarn (2.0.0-cdh4.2.1). But there is no log after execution. The following command is used to run a spark example. But logs are not found in the history server as in …

Read More