hadoop: difference between 0 reducer and identity reducer?

hadoop mapreduce
kee picture kee · May 17, 2012 · Viewed 17.1k times · Source

I am just trying to confirm my understanding of difference between 0 reducer and identity reducer.

  • 0 reducer means reduce step will be skipped and mapper output will be the final out
  • Identity reducer means then shuffling/sorting will still take place?

Answer

David Gruzman picture David Gruzman · May 17, 2012

You understanding is correct. I would define it as following: If you do not need sorting of map results - you set 0 reduced,and the job is called map only.
If you need to sort the mapping results, but do not need any aggregation - you choose identity reducer.
And to complete the picture we have a third case : we do need aggregation and, in this case we need reducer.

Related questions

What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

I am getting: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask While trying to make a copy of a partitioned table using the commands in the hive console: CREATE TABLE copy_table_name LIKE table_…

Read More
Setting the number of map tasks and reduce tasks

I am currently running a job I fixed the number of map task to 20 but and getting a higher number. I also set the reduce task to zero but I am still getting a number other than zero. The total …

Read More
Container is running beyond memory limits

In Hadoop v1, I have assigned each 7 mapper and reducer slot with size of 1GB, my mappers & reducers runs fine. My machine has 8G memory, 8 processor. Now with YARN, when run the same application on the same machine, I …

Read More