terminating a spark step in aws

hadoop amazon-web-services apache-spark emr
Daniel Imberman picture Daniel Imberman · Jan 26, 2016 · Viewed 8.2k times · Source

I want to set up a series of spark steps on an EMR spark cluster, and terminate the current step if it's taking too long. However, when I ssh into the master node and run hadoop jobs -list, the master node seems to believe that there is no jobs running. I don't want to terminate the cluster, because doing so would force me to buy a whole new hour of whatever cluster I'm running. Can anyone please help me terminate a spark-step in EMR without terminating the entire cluster?

Answer

Erik Schmiegelow picture Erik Schmiegelow · Jan 26, 2016

That's easy:

yarn application -kill [application id]

you can list your running applications with

yarn application -list

Related questions

How to set a custom environment variable in EMR to be available for a spark Application

I need to set a custom environment variable in EMR to be available when running a spark application. I have tried adding this: ... --configurations '[ { "Classification": "spark-env", "Configurations": [ { "Classification": "export", "Configurations": [], "Properties": { "SOME-ENV-VAR": "qa1" } } ], "Properties": {} } ]' ... and also tried to …

Read More
Hadoop/Hive : Loading data from .csv on a local machine

As this is coming from a newbie... I had Hadoop and Hive set up for me, so I can run Hive queries on my computer accessing data on AWS cluster. Can I run Hive queries with .csv data stored on …

Read More
Deleting file/folder from Hadoop

I'm running an EMR Activity inside a Data Pipeline analyzing log files and I get the following error when my Pipeline fails: Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://10.208.42.127:9000/home/hadoop/temp-output-s3copy already exists …

Read More