Spark Kill Running Application

apache-spark yarn pyspark
B.Mr.W. picture B.Mr.W. · Apr 10, 2015 · Viewed 166.4k times · Source

I have a running Spark application where it occupies all the cores where my other applications won't be allocated any resource.

I did some quick research and people suggested using YARN kill or /bin/spark-class to kill the command. However, I am using CDH version and /bin/spark-class doesn't even exist at all, YARN kill application doesn't work either.

enter image description here

Can anyone with me with this?

Answer

Gérald Reinhart picture Gérald Reinhart · May 15, 2015
  • copy past the application Id from the spark scheduler, for instance application_1428487296152_25597
  • connect to the server that have launch the job
  • yarn application -kill application_1428487296152_25597

Related questions

Do exit codes and exit statuses mean anything in spark?

I see exit codes and exit statuses all the time when running spark on yarn: Here are a few: CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM ...failed 2 times due to AM Container for application_1431523563856_0001_000002 exited with exitCode: 10... ...Exit status: 143. Diagnostics: Container killed on …

Read More
Exception: java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. in spark

I am new apache-spark. I have tested some application in spark standalone mode.but I want to run application yarn mode.I am running apache-spark 2.1.0 in windows.Here is My code c:\spark>spark-submit2 --master yarn --deploy-mode client --executor-cores 4 …

Read More
WARN cluster.YarnScheduler: Initial job has not accepted any resources

Any spark jobs that I run will fail with the following error message 17/06/16 11:10:43 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources Spark version is 1.6, …

Read More