Chaining multiple mapreduce tasks in Hadoop streaming

python hadoop mapreduce hadoop-plugins
Varadharajan Mukundan picture Varadharajan Mukundan · Jan 7, 2011 · Viewed 7.3k times · Source

I am in scenario where I have two mapreduce jobs. I am more comfortable with python and planning to use it for writing mapreduce scripts and use hadoop streaming for the same. is there a convenient to chain both the jobs following form when hadoop streaming is used?

Map1 -> Reduce1 -> Map2 -> Reduce2

I've heard a lot of methods to accomplish this in java, But i need something for Hadoop streaming.

Answer

cwensel picture cwensel · Jan 7, 2011

Here is a great blog post on how to use Cascading and Streaming. http://www.xcombinator.com/2009/11/18/how-to-use-cascading-with-hadoop-streaming/

The value here is you can mix java (Cascading query flows) with your custom streaming operations in the same app. I find this much less brittle than other methods.

Note, the Cascade object in Cascading allows you to chain multiple Flows (via the above blog post your Streaming job would become a MapReduceFlow).

Disclaimer: I'm the author of Cascading

Related questions

Hadoop Streaming Job failed error in python

From this guide, I have successfully run the sample exercise. But on running my mapreduce job, I am getting the following error ERROR streaming.StreamJob: Job not Successful! 10/12/16 17:13:38 INFO streaming.StreamJob: killJob... Streaming Job Failed! Error from the log file …

Read More
Running a job using hadoop streaming and mrjob: PipeMapRed.waitOutputThreads(): subprocess failed with code 1

Hey I'm fairly new to the world of Big Data. I came across this tutorial on http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/ It describes in detail of how to run MapReduce job using mrjob both locally and on Elastic Map Reduce. …

Read More
Python MapReduce Hadoop Streaming Job that requires multiple input files?

I have two files in my cluster File A and File B with the following data - File A #Format: #Food Item | Is_A_Fruit (BOOL) Orange | Yes Pineapple | Yes Cucumber | No Carrot | No Mango | Yes File B #Format: #Food …

Read More