Main Class Not Found when Compiling Hadoop Sample Program

Question

Hi i'm new to Hadoop and just started learning a couple days ago. I just followed the instructions from Digital Ocean to setup a Hadoop cluster. Afterwards I just tried a simple example program called WordCount from the Hadoop docs.

My hadoop version is 2.5.1 which is the same version with what is used on the tutorial, and it's running on Ubuntu Precise. I'm ensuring that I've done the proper setup as the tutorial said. Here's the end of my ~/.bashrc contents.

        ...
#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
export HADOOP_PREFIX=/usr/local/hadoop
#HADOOP VARIABLES END

Also, I checked on my java home config and the result is like below

sudo update-alternatives --config java
There are 3 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      auto mode
  1            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      manual mode
  2            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1051      manual mode
* 3            /usr/lib/jvm/java-7-oracle/jre/bin/java          1         manual mode

So i changed all JAVA_HOME value both in bashrc and hadoop-env.sh files to /usr/lib/jvm/java-7-oracle . I'm ensuring as well that the Dfs and Yarn are both started.

However, when i compile the WordCount.java using this command

hadoop com.sun.tools.javac.Main WordCount.java 

Nothing is going my way. I got this error. Note that i'm using Hadoop command instead bin/hadoop as the command is working properly since it was defined in bashrc file.

Error: Could not find or load main class com.sun.tools.javac.Main

What is the possible cause of this error and how to get rid of this? It might be java classpath issue i think, but i'm still not be able to figure out the detail. Every workarounds regarding this problem i got are about executing java or javac command, not hadoop command.

I just want to get the sample program working first, before getting started to learn how it works. Any help would be appreciated. Thanks..

Answer 1

The Apache Hadoop tutorial assumes that the environmental variables are set as follows:

export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar

Perhaps the Digital Ocean hadoop tutorial, which I also followed, ought to recommend adding those two latter variables to the ~/.bashrc so that it ends up looking like this:

#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
#HADOOP VARIABLES END

It worked for my installation. See the new compiled class files listed in the output:

Before:

ubuntu@mail:/usr/local/hadoop$ ls
bin  include  lib      LICENSE.txt  NOTICE.txt  README.txt  share              WordCount.java
etc  input    libexec  logs         output      sbin        WordCount_classes

After:

ubuntu@mail:/usr/local/hadoop$ bin/hadoop com.sun.tools.javac.Main WordCount.java
ubuntu@mail:/usr/local/hadoop$ ls
bin      input    LICENSE.txt  output      share              WordCount$IntSumReducer.class
etc      lib      logs         README.txt  WordCount.class    WordCount.java
include  libexec  NOTICE.txt   sbin        WordCount_classes  WordCount$TokenizerMapper.class

Another helpful resource was this:

http://ubuntuforums.org/archive/index.php/t-634996.html Append the following lines to the opened .bashrc file, save it and close: export JAVA_HOME="/usr/lib/jvm/java-6-sun-1.6.0.22" export PATH=$PATH:$JAVA_HOME/bin issue the following command in the terminal: source $HOME/.bashrc

please refer this blog post for more info (http://sureshatt.blogspot.com/2011/01/easiest-way-to-install-java-in-ubuntu.html)