Hadoop on windows server

c# windows hadoop mapreduce cluster-computing
Luca Martinetti picture Luca Martinetti · Jan 22, 2009 · Viewed 9.3k times · Source

I'm thinking about using hadoop to process large text files on my existing windows 2003 servers (about 10 quad core machines with 16gb of RAM)

The questions are:

  1. Is there any good tutorial on how to configure an hadoop cluster on windows?

  2. What are the requirements? java + cygwin + sshd ? Anything else?

  3. HDFS, does it play nice on windows?

  4. I'd like to use hadoop in streaming mode. Any advice, tool or trick to develop my own mapper / reducers in c#?

  5. What do you use for submitting and monitoring the jobs?

Thanks

Answer

bradheintz picture bradheintz · Feb 20, 2009

From the Hadoop documentation:

Win32 is supported as a development platform. Distributed operation has not been well tested on Win32, so it is not supported as a production platform.

Which I think translates to: "You're on your own."

That said, there might be hope if you're not queasy about installing Cygwin and a Java shim, according to the Getting Started page of the Hadoop wiki:

It is also possible to run the Hadoop daemons as Windows Services using the Java Service Wrapper (download this separately). This still requires Cygwin to be installed as Hadoop requires its df command.

I guess the bottom line is that it doesn't sound impossible, but you'd be swimming upstream all the way. I've done a few Hadoop installs (on Linux for production, Mac for dev) now, and I wouldn't bother with Windows when it's so straightforward on other platforms.

Related questions

How to delete a file after checking whether it exists

How can I delete a file in C# e.g. C:\test.txt, although apply the same kind of method like in batch files e.g. if exist "C:\test.txt" delete "C:\test.txt" else return nothing (ignore)

Read More
How to get a path to the desktop for current user in C#?

How do I get a path to the desktop for current user in C#? The only thing I could find was the VB.NET-only class SpecialDirectories, which has this property: My.Computer.FileSystem.SpecialDirectories.Desktop How can I do this …

Read More
How do I start a process from C#?

How do I start a process, such as launching a URL when the user clicks a button?

Read More