Error in pig while loading data

hadoop apache-pig
Hardik Barot picture Hardik Barot · Jan 23, 2014 · Viewed 13.3k times · Source

I am using ubuntu 12.02 32bit and have installed hadoop2.2.0 and pig 0.12 successfully. Hadoop runs properly on my system.

However, whenever I run this command :

data = load 'atoz.csv' using PigStorage(',')  as (aa1:int, bb1:int, cc1:int, dd1:chararray);            
dump data;

I'm getting the following error :

ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error whiletrying to run jobs.java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected.

Here is the full stacktrace :

> 2014-01-23 10:41:44,998 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 1 map-reduce job(s) waiting for submission.
>             2014-01-23 10:41:45,000 [Thread-9] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
> Metrics with processName=JobTracker, sessionId= - already initialized
>             2014-01-23 10:41:45,001 [Thread-9] ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while
> trying to run jobs.
>             java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
>             at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>             at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>             at java.security.AccessController.doPrivileged(Native Method)
>             at javax.security.auth.Subject.doAs(Subject.java:415)
>             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>             at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
>             at java.lang.Thread.run(Thread.java:724)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>             2014-01-23 10:41:45,498 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
>             2014-01-23 10:41:45,502 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job null has failed! Stop running all dependent jobs
>             2014-01-23 10:41:45,503 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
>             2014-01-23 10:41:45,507 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> recreate exception from backend error: Unexpected System Error
> Occured: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
>             at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>             at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>             at java.security.AccessController.doPrivileged(Native Method)
>             at javax.security.auth.Subject.doAs(Subject.java:415)
>             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>             at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
>             at java.lang.Thread.run(Thread.java:724)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
2014-01-23 10:41:45,507 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s)
> failed!
>             2014-01-23 10:41:45,507 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode.
> Stats reported below may be incomplete
>             2014-01-23 10:41:45,508 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion    PigVersion    UserId    StartedAt    FinishedAt    Features
>             2.2.0    0.10.1    hardik    2014-01-23 10:41:44    2014-01-23 10:41:45    UNKNOWN
 Failed!
Failed Jobs:
JobId    Alias    Feature    Message    Outputs
N/A    aatoz    MAP_ONLY    Message: Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
>             at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>             at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>             at java.security.AccessController.doPrivileged(Native Method)
>             at javax.security.auth.Subject.doAs(Subject.java:415)
>             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>             at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
>             at java.lang.Thread.run(Thread.java:724)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>             file:/tmp/temp1979716161/tmp-189979005,
Input(s):
Failed to read data from "file:///home/hardik/pig10/bin/input/atoz.csv"
Output(s):
             Failed to produce result in "file:/tmp/temp1979716161/tmp-189979005"
Job DAG:
null
2014-01-23 10:41:45,509 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 Failed! 2014-01-23 10:41:45,510 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator
> for alias aatoz
>             Details at logfile: /home/hardik/pig10/bin/pig_1390453192689.log
>     </i>

Answer

Michal picture Michal · Feb 13, 2014

Apache Pig 0.12.0 expects an older version of Hadoop by default. You must recompile Pig for Hadoop 2.2.0 and replace two jars with new pig-0.12.1-SNAPSHOT.jar and pig-0.12.1-SNAPSHOT-withouthadoop.jar.

For recompilation unpack the pig archive, go to the directory "pig-0.12.0" and just run:

ant clean jar-all -Dhadoopversion=23

Related questions

Difference between Pig and Hive? Why have both?

My background - 4 weeks old in the Hadoop world. Dabbled a bit in Hive, Pig and Hadoop using Cloudera's Hadoop VM. Have read Google's paper on Map-Reduce and GFS (PDF link). I understand that- Pig's language Pig Latin is a …

Read More
When to use Hadoop, HBase, Hive and Pig?

What are the benefits of using either Hadoop or HBase or Hive ? From my understanding, HBase avoids using map-reduce and has a column oriented storage on top of HDFS. Hive is a sql-like interface for Hadoop and HBase. I would …

Read More
PIG how to count a number of rows in alias

I did something like this to count the number of rows in an alias in PIG: logs = LOAD 'log' logs_w_one = foreach logs generate 1 as one; logs_group = group logs_w_one all; logs_count = foreach logs_group generate …

Read More