combiner and reducer can be different?

mapreduce reducers combiners
kee picture kee · Jul 31, 2012 · Viewed 25.8k times · Source

In many MapReduce programs, I see a reducer being used as a combiner as well. I know this is because of the specific nature of those programs. But I am wondering if they can be different.

Answer

Binary Nerd picture Binary Nerd · Jul 31, 2012

Yes, a combiner can be different to the Reducer, although your Combiner will still be implementing the Reducer interface. Combiners can only be used in specific cases which are going to be job dependent. The Combiner will operate like a Reducer, but only on the subset of the Key/Values output from each Mapper.

One constraint that your Combiner will have, unlike a Reducer, is that the input/output key and value types must match the output types of your Mapper.

Related questions

“Combiner" Class in a mapreduce job

A Combiner runs after the Mapper and before the Reducer,it will receive as input all data emitted by the Mapper instances on a given node. then emits output to the Reducers. And also,If a reduce function is both …

Read More
Number of reducers in hadoop

I was learning hadoop, I found number of reducers very confusing : 1) Number of reducers is same as number of partitions. 2) Number of reducers is 0.95 or 1.75 multiplied by (no. of nodes) * (no. of maximum containers per node). 3) Number of reducers is …

Read More
saving json data in hdfs in hadoop

I have the following Reducer class public static class TokenCounterReducer extends Reducer<Text, Text, Text, Text> { public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException { JSONObject jsn = new JSONObject(); for (Text value : …

Read More