Fault Tolerance in MapReduce

mapreduce distributed-computing fault-tolerance
Chander Shivdasani picture Chander Shivdasani · Apr 28, 2011 · Viewed 11k times · Source

I was reading about Hadoop and how fault tolerant it is. I read the HDFS and read how failure of master and slave nodes can be handled. However, i couldnt find any document that mentions how the mapreduce performs fault tolerance. Particularly, what happens when the Master node containing Job Tracker goes down or any of the slave nodes goes down?

If anyone can point me to some links and references that explains this in detail.

Answer

Chander Shivdasani picture Chander Shivdasani · Apr 28, 2011

Fault Tolerance of MapReduce layer depends on the hadoop version. For versions before hadoop.0.21, no checkpointing was done and failure of JobTracker would lead to loss of data.

However, versions starting hadoop.0.21, checkpointing was added where JobTracker records its progress in a file. When a JobTracker starts up, it looks for such data, so that it can restart work from where it left off.

Related questions

Change File Split size in Hadoop

I have a bunch of small files in an HDFS directory. Although the volume of the files is relatively small, the amount of processing time per file is huge. That is, a 64mb file, which is the default split size …

Read More
What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

I am getting: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask While trying to make a copy of a partitioned table using the commands in the hive console: CREATE TABLE copy_table_name LIKE table_…

Read More
Java8: HashMap<X, Y> to HashMap<X, Z> using Stream / Map-Reduce / Collector

I know how to "transform" a simple Java List from Y -> Z, i.e.: List<String> x; List<Integer> y = x.stream() .map(s -> Integer.parseInt(s)) .collect(Collectors.toList()); Now I'd …

Read More