Difference between Fork/Join and Map/Reduce

mapreduce fork-join
hotzen picture hotzen · Mar 29, 2010 · Viewed 9k times · Source

What is the key difference between Fork/Join and Map/Reduce?

Do they differ in the kind of decomposition and distribution (data vs. computation)?

Answer

tucuxi picture tucuxi · Mar 29, 2010

One key difference is that F-J seems to be designed to work on a single Java VM, while M-R is explicitly designed to work on a large cluster of machines. These are very different scenarios.

F-J offers facilities to partition a task into several subtasks, in a recursive-looking fashion; more tiers, possibility of 'inter-fork' communication at this stage, much more traditional programming. Does not extend (at least in the paper) beyond a single machine. Great for taking advantage of your eight-core.

M-R only does one big split, with the mapped splits not talking between each other at all, and then reduces everything together. A single tier, no inter-split communication until reduce, and massively scalable. Great for taking advantage of your share of the cloud.

Related questions

What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

I am getting: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask While trying to make a copy of a partitioned table using the commands in the hive console: CREATE TABLE copy_table_name LIKE table_…

Read More
Java8: HashMap<X, Y> to HashMap<X, Z> using Stream / Map-Reduce / Collector

I know how to "transform" a simple Java List from Y -> Z, i.e.: List<String> x; List<Integer> y = x.stream() .map(s -> Integer.parseInt(s)) .collect(Collectors.toList()); Now I'd …

Read More
Good MapReduce examples

I couldn't think of any good examples other than the "how to count words in a long text with MapReduce" task. I found this wasn't the best example to give others an impression of how powerful this tool can be. …

Read More