what is the basic difference between jobconf and job?

hadoop mapreduce bigdata
user1585111 picture user1585111 · Aug 23, 2013 · Viewed 13.4k times · Source

hi i wanted to know the basic difference between jobconf and job objects,currently i am submitting my job like this

JobClient.runJob(jobconf);

i saw other way of submitting jobs like this

Configuration conf = getConf();
Job job = new Job(conf, "secondary sort");
job.waitForCompletion(true);     
return 0;

and how can i specify the sortcomparator class for the job using jobconf?

can any one explain me this concept?

Answer

jkovacs picture jkovacs · Aug 23, 2013

In short: JobConf and everything else in the org.apache.hadoop.mapred package is part of the old API used to write hadoop jobs, Job and everything in the org.apache.hadoop.mapreduce package is part of the new and preferred API to write hadoop jobs. Both APIs generally provide equivalent core functionality.

If you're new to hadoop just start using the new API (i.e. Job and Configuration instead of JobConf). Make sure to not import anything from the mapred package. When you find examples on the internet using the old API you can use this presentation or this guide to translate it to the new API.

Related questions

Hive ParseException - cannot recognize input near 'end' 'string'

I am getting the following error when trying to create a Hive table from an existing DynamoDB table: NoViableAltException(88@[]) at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:9123) at org.apache.hadoop.hive.ql.parse.HiveParser.…

Read More
how to sort word count by value in hadoop?

hi i wanted to learn how to sort the word count by value in hadoop.i know hadoop takes of sorting keys, but not by values. i know to sort the values we must have a partitioner,groupingcomparator and a …

Read More
Number of reducers in hadoop

I was learning hadoop, I found number of reducers very confusing : 1) Number of reducers is same as number of partitions. 2) Number of reducers is 0.95 or 1.75 multiplied by (no. of nodes) * (no. of maximum containers per node). 3) Number of reducers is …

Read More