Top "Hadoop2" questions

How to optimize shuffling/sorting phase in a hadoop job

I'm doing some data preparation using a single node hadoop job. The mapper/combiner in my job outputs many keys (…

hadoop mapreduce hadoop2

Standard practices for logging in MapReduce jobs

I'm trying to find the best approach for logging in MapReduce jobs. I'm using slf4j with log4j appender …

java hadoop mapreduce hadoop2 mapr

Is there the equivalent for a `find` command in `hadoop`?

I know that from the terminal, one can do a find command to find files such as : find . -type d …

hadoop terminal hdfs hadoop2

could only be replicated to 0 nodes instead of minReplication (=1). There are 4 datanode(s) running and no node(s) are excluded in this operation

I don't know how to fix this error: Vertex failed, vertexName=initialmap, vertexId=vertex_1449805139484_0001_1_00, diagnostics=[Task failed, taskId=task_1449805139484_0001_1_00_000003, diagnostics=[…

hadoop hdfs yarn hadoop2 apache-tez

How do you retrieve the replication factor info in Hdfs files?

I have set the replication factor for my file as follows: hadoop fs -D dfs.replication=5 -copyFromLocal file.txt /user/…

hadoop replication hadoop2

Hadoop Error - All data nodes are aborting

I am using Hadoop 2.3.0 version. Sometimes when I execute the Map reduce job, the below errors will get displayed. 14/08/10 12:14:59 INFO …

hadoop mapreduce hdfs yarn hadoop2

copyFromLocal: `/user/hduser/gutenberg': No such file or directory

I have followed the guide of michael-noll so far but got stuck here. hduser@ubuntu:/usr/local/hadoop$ bin/hadoop …

hadoop hadoop2

Permission Denied error while running start-dfs.sh

I am getting this error while performing start-dfs.sh Starting namenodes on [localhost] pdsh@Gaurav: localhost: rcmd: socket: Permission denied …

sockets hadoop hdfs yarn hadoop2

What is Memory reserved on Yarn

I managed to launch a spark application on Yarn. However emory usage is kind of weird as you can see …

hadoop apache-spark yarn hadoop2

Difference between a ring buffer and a queue

What is the difference between the ring (circular) buffer and a queue? Both support FIFO so in what scenarios I …

hadoop data-structures hadoop2