Top "Yarn" questions

Why does spark-shell --master yarn-client fail (yet pyspark --master yarn seems to work)?

I'm trying to run the spark shell on my Hadoop cluster via Yarn. I use Hadoop 2.4.1 Spark 1.0.0 My Hadoop cluster …

hdfs apache-spark yarn

AM Container is running beyond virtual memory limits

I was playing with distributed shell application (hadoop-2.0.0-cdh4.1.2). This is the error I'm receiving at the moment. 13/01/01 17:09:09 INFO distributedshell.…

hadoop cloudera yarn

How to exit spark-submit after the submission

When submitting spark streaming program using spark-submit(YARN mode) it keep polling the status and never exit Is there any …

apache-spark yarn

Spark resources not fully allocated on Amazon EMR

I'm trying to maximize cluster usage for a simple task. Cluster is 1+2 x m3.xlarge, runnning Spark 1.3.1, Hadoop 2.4, Amazon AMI 3.7 …

apache-spark yarn emr

Spark submit to yarn as a another user

Is it possible to submit a spark job to a yarn cluster and choose, either with the command line or …

hadoop apache-spark yarn kerberos

Aggregate Resource Allocation for a job in YARN

I am new to Hadoop. When i run a job, i see the aggregate resource allocation for that job as 251248654 …

hadoop yarn

Spark on yarn mode end with "Exit status: -100. Diagnostics: Container released on a *lost* node"

I am trying to load a database with 1TB data to spark on AWS using the latest EMR. And the …

apache-spark yarn emr

could only be replicated to 0 nodes instead of minReplication (=1). There are 4 datanode(s) running and no node(s) are excluded in this operation

I don't know how to fix this error: Vertex failed, vertexName=initialmap, vertexId=vertex_1449805139484_0001_1_00, diagnostics=[Task failed, taskId=task_1449805139484_0001_1_00_000003, diagnostics=[…

hadoop hdfs yarn hadoop2 apache-tez

Hadoop: specify yarn queue for distcp

On our cluster we have set up dynamic resource pools. The rules are set so that first yarn will look …

hadoop yarn distcp

Exception: java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. in spark

I am new apache-spark. I have tested some application in spark standalone mode.but I want to run application yarn …

hadoop apache-spark pyspark yarn