Why isn't Hadoop implemented using MPI?

tcp hadoop protocol-buffers mpi distributed-computing
artif picture artif · Jan 4, 2011 · Viewed 19.3k times · Source

Correct me if I'm wrong, but my understanding is that Hadoop does not use MPI for communication between different nodes.

What are the technical reasons for this?

I could hazard a few guesses, but I do not know enough of how MPI is implemented "under the hood" to know whether or not I'm right.

Come to think of it, I'm not entirely familiar with Hadoop's internals either. I understand the framework at a conceptual level (map/combine/shuffle/reduce and how that works at a high level) but I don't know the nitty gritty implementation details. I've always assumed Hadoop was transmitting serialized data structures (perhaps GPBs) over a TCP connection, eg during the shuffle phase. Let me know if that's not true.

Answer

user510306 picture user510306 · Jan 4, 2011

One of the big features of Hadoop/map-reduce is the fault tolerance. Fault tolerance is not supported in most (any?) current MPI implementations. It is being thought about for future versions of OpenMPI.

Sandia labs has a version of map-reduce which uses MPI, but it lacks fault tolerance.

Related questions

How can I connect to Android with ADB over TCP?

I am attempting to debug an application on a Motorola Droid, but I am having some difficulty connecting to the device via USB. My development server is a Windows 7 64-bit VM running in Hyper-V, and so I cannot connect directly …

Read More
Who is listening on a given TCP port on Mac OS X?

On Linux, I can use netstat -pntl | grep $PORT or fuser -n tcp $PORT to find out which process (PID) is listening on the specified TCP port. How do I get the same information on Mac OS X?

Read More
java.net.ConnectException: Connection refused

I'm trying to implement a TCP connection, everything works fine from the server's side but when I run the client program (from client computer) I get the following error: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native …

Read More