Top "Yarn" questions

YARN (Yet Another Resource Negotiator) is a key component of second generation Apache Hadoop infrastructure.

How to set amount of Spark executors?

How could I configure from Java (or Scala) code amount of executors having SparkConfig and SparkContext? I see constantly 2 executors. …

java scala cluster-computing apache-spark yarn
How to prevent Spark Executors from getting Lost when using YARN client mode?

I have one Spark job which runs fine locally with less data but when I schedule it on YARN to …

apache-spark yarn
Application report for application_ (state: ACCEPTED) never ends for Spark Submit (with Spark 1.2.0 on YARN)

I am running kinesis plus spark application https://spark.apache.org/docs/1.2.0/streaming-kinesis-integration.html I am running as below command …

apache-spark yarn amazon-emr amazon-kinesis
Yarn MapReduce Job Issue - AM Container launch error in Hadoop 2.3.0

I have setup a 2 node cluster of Hadoop 2.3.0. Its working fine and I can successfully run distributedshell-2.2.0.jar example. But …

java hadoop mapreduce yarn
FetchFailedException or MetadataFetchFailedException when processing big data set

When I run the parsing code with 1 GB dataset it completes without any error. But, when I attempt 25 gb of …

apache-spark yarn
Spark yarn cluster vs client - how to choose which one to use?

The spark docs have the following paragraph that describes the difference between yarn client and yarn cluster: There are two …

apache-spark yarn
Why does Hadoop report "Unhealthy Node local-dirs and log-dirs are bad"?

I am trying to setup a single-node Hadoop 2.6.0 cluster on my PC. On visiting http://localhost:8088/cluster, I find that …

hadoop yarn
spark on yarn, Container exited with a non-zero exit code 143

I am using HDP 2.5, running spark-submit as yarn cluster mode. I have tried to generate data using dataframe cross join. …

apache-spark hive yarn hortonworks-data-platform
Which cluster type should I choose for Spark?

I am new to Apache Spark, and I just learned that Spark supports three types of cluster: Standalone - meaning …

apache-spark yarn mesos apache-spark-standalone
Spark runs on Yarn cluster exitCode=13:

I am a spark/yarn newbie, run into exitCode=13 when I submit a spark job on yarn cluster. When the …

scala hadoop apache-spark yarn