Hadoop on Windows Building/ Installation Error

windows hadoop msbuild building
Cristian Ciobotea picture Cristian Ciobotea · Jul 8, 2015 · Viewed 8.7k times · Source

I am trying to install Apache Hadoop 2.7.1 on Windows x64 (8.1 and Server 2012 R2), but I am failing at goal:

    [INFO] Apache Hadoop Common ............................... FAILURE [ 37.925 s]

In the past days, I think I followed the steps of all tutorials, like HadoopOnWindows, or this one. I did all the prerequisites, but as I Googled, the relevant ones for my problem are the following:

  • Set path to MSBuild.exe like: C:\Windows\Microsoft.NET\Framework64\v4.0.30319; in Path
  • Set path to CMake
  • Install Microsoft SDK 7.1 and Run with its Microsoft SDK 7.1 CMD
  • Tried to build: hadoop-common\src\main\winutils\libwinutils.sln and hadoop-common\src\main\winutils\winutils.sln with Visual Studio 2013, but building fails due to compiling errors. Visual Studio 2010 doesn't even want to open them. I do not know how to correct the compile errors (in case this is the problem)

Other ppl, similar problem
Some of the similar topics I am following are these: one, two, three, four. I mainly tried all the suggestions, but it still breaks.

Error 

    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 01:11 min
    [INFO] Finished at: 2015-07-08T15:18:29+02:00
    [INFO] Final Memory: 69M/269M
    [INFO] ------------------------------------------------------------------------
    [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (c
    ompile-ms-winutils) on project hadoop-common: Command execution failed. Process
    exited with an error: 1 (Exit value: 1) -> [Help 1]
    [ERROR]
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit
    ch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please rea
    d the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
    xception
    [ERROR]
    [ERROR] After correcting the problems, you can resume the build with the command

    [ERROR]   mvn <goals> -rf :hadoop-common

Help
I have no idea what to do next, I feel like I tried everything. Please help me solve this error and continue with the Hadoop install on Windows.

UPDATE 1 Looking on the error stack more closely, I saw that the file Microsoft.Cpp.props was not found. Moreover, the entire folder Program Files (x86)/MSBuild was missing. Thus, I installed Visual Studio 2010 and it fixed this problem.

Of course, a new problem arise. I am able to build libwinutils, but not winutils. I get many LNK2001 errors:

Error   48  error LNK1120: 18 unresolved externals  C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\x64\Release\winutils.exe  winutils
Error   36  error LNK2001: unresolved external symbol BuildServiceSecurityDescriptor    C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj   winutils
Error   32  error LNK2001: unresolved external symbol ChownImpl C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj   winutils
Error   40  error LNK2001: unresolved external symbol CreateEnvironmentBlock    C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj  winutils
Error   44  error LNK2001: unresolved external symbol CreateLogonTokenForUser   C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj  winutils
Error   41  error LNK2001: unresolved external symbol DestroyEnvironmentBlock   C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj  winutils
Error   37  error LNK2001: unresolved external symbol EnableImpersonatePrivileges   C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj   winutils
Error   34  error LNK2001: unresolved external symbol GetSecureJobObjectName    C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj   winutils
Error   38  error LNK2001: unresolved external symbol KillTask  C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj   winutils
Error   43  error LNK2001: unresolved external symbol LoadUserProfileForLogon   C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj  winutils
Error   35  error LNK2001: unresolved external symbol LogDebugMessage   C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj   winutils
Error   46  error LNK2001: unresolved external symbol LookupKerberosAuthenticationPackageId C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj  winutils
Error   31  error LNK2001: unresolved external symbol MIDL_user_allocate    C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\hadoopwinutilsvc_s.obj    winutils
Error   30  error LNK2001: unresolved external symbol MIDL_user_free    C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\hadoopwinutilsvc_s.obj    winutils
Error   47  error LNK2001: unresolved external symbol RegisterWithLsa   C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj  winutils
Error   33  error LNK2001: unresolved external symbol SplitStringIgnoreSpaceW   C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj   winutils
Error   42  error LNK2001: unresolved external symbol UnloadProfileForLogon C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj  winutils
Error   45  error LNK2001: unresolved external symbol UnregisterWithLsa C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj  winutils
Error   39  error LNK2001: unresolved external symbol wsceConfigRelativePath    C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj   winutils

UPDATE 2 The answer from @tiho solved the problem mentioned above (and I am very thankful, as it took me 4 days already). Now, a fresh new problem, at goal:

[INFO] Apache Hadoop KMS .................................. FAILURE [  1.531 s]

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project hadoop-kms: An Ant BuildException has occured: java.net.UnknownHostException: archive.apache.org
[ERROR] around Ant part ...<get dest="downloads/apache-tomcat-6.0.41.tar.gz" skipexisting="true" verbose="true" src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.41/bin/apache tomcat-6.0.41.tar.gz"/>... @ 5:182 in C:\hadoop-2.7.1-src\hadoop-common project\hadoop-kms\target\antrun\build-main.xml: Unknown host archive.apache.org.

I found a lame work-around to the problem above. I manually downloaded it, added to the folders:

C:\hadoop-2.7.1-src\hadoop-common-project\hadoop-kms\downloads
C:\hadoop-2.7.1-src\hadoop-hdfs-project\hadoop-hdfs-httpfs\downloads
and deleted from the files:
C:\hadoop-2.7.1-src\hadoop-common-project\hadoop-kms\target\antrun\build-main.xml
C:\hadoop-2.7.1-src\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\antrun\build-main.xml
the following lines of code:

<mkdir dir="downloads"/>
  <get dest="downloads/apache-tomcat-6.0.41.tar.gz" skipexisting="true" verbose="true" src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.41/bin/apache-tomcat-6.0.41.tar.gz"/>

Note that I am also using proxy, I have set the proxy in the settings.xml, and set PATH to MAVEN_OPTS as suggested in the same answer from @tiho. Is this the problem? besides the options suggested there, I have also added password and username to the MAVEN_OPTS.

Note that I can access: http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.41/bin/apache-tomcat-6.0.41.tar.gz using my browser.

But I am still curious how to solve the issue correctly, because this problem continues to appear, and I need to do the same work-around again.

Update 3 That being said, the happiest moment of this week:

[INFO] ------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------
[INFO] Total time: 21:05 min
[INFO] Finished at: 2015-07-10T11:33:17+02:00
[INFO] Final Memory: 228M/672M
[INFO] ------------------------------------------------

Answer

Romeu Flores picture Romeu Flores · Aug 14, 2015

I've managed to build it with Visual Studio 2015 community edition.

Here's how I've built it:

My environment

Here's my shopping list:

  • Windows 10
  • JDK 1.8.0_51
  • Maven 3.3.3
  • Findbugs 1.3.9 (I haven't used this)
  • ProtocolBuffer 2.5.0 (I didn't pick the latest and greatest here - it has to be 2.5.0)
  • CMake 3.3.0
  • Visual Studio 2015 Community Edition
  • GnuWin32 0.6.3 - a bit painful to install but so is cygwin
  • zlib 1.2.8
  • internet connection

Windows System Environment variables

  • JAVA_HOME = "C:\Program Files\Java\jdk1.8.0_51"
  • MAVEN_HOME=c:\apache-maven-3.3.3

(make sure you point the above to your JDK version and maven installation)

I appended the following to my windows system environment Path variable:

;%MAVEN_HOME%\bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;c:\zlib

The weird "C:\Windows\Microsoft.NET\Framework64\v4.0.30319" path is the location of MSBuild.exe, which is required during the build process.

Protoc Buffers 2.5.0

Oh no, another unix/linux only build? I've downloaded the google package named protoc-2.5.0-win32.zip. Then extracted the binary file (protoc.exe) to c:\windows\system32 - just a lazy way to put it on the path.

I'm not 100% sure of the effect of having a win32 component for this win64 build. But: "Hadoop 0.23+ requires the protocol buffers JAR (protobufs.jar) to be on the classpath of both clients and servers; the native binaries are required to compile this and later versions of Hadoop." - http://wiki.apache.org/hadoop/ProtocolBuffers.

So I understand the win32 executable is used only during the build process (the jar equivalent should be packaged in the build).

If it is used in any way to compile native code, we may have left with some pointers out of order. I'll come back to this when I can.

Tweaking the Hadoop sources

Well, this was necessary to allow to build to execute. It shouldn't affect the quality of the build itself, but let's keep in mind the result is an unofficial, unsupported, use at your own risk hadoop, intended for a development environment.

Migrating VS projects

The following files need to be open with Visual Studio 2015:

\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj \hadoop-common-project\hadoop-common\src\main\native\native.vcxproj

Visual Studio will complain of them being of an old version. All you have to do is to save all and close.

Enabling cmake VS 2015 project generation for hdfs

On the line 441 of \hadoop-hdfs-project\hadoop-hdfs\pom.xml, edit the else value as the following:

<condition property="generator" value="Visual Studio 10" else="Visual Studio 14 2015 Win64">

(the "value" value applies to win32 - you may want to edit it if building for win32).

Building it

You should try and find on windows the "Development Command Prompt for VS2015". I'm still wondering what is so special about this, but the fact is that it will only work with that.

More Environment variables Those should be done on the command prompt:

set Platform=x64

set ZLIB_HOME=C:\zlib\include (unlike the official instructions, this should be pointing to the include folder).

Finally building it

Go to the hadoop source folder and issue:

mvn package -Pdist,native-win -DskipTests -Dtar

What next?

Follow the official docs to get your hadoop instance configured and up and running.

I'll try to keep a link with for the binaries on my blog:

http://kplitzkahran.blogspot.co.uk/2015/08/hadoop-271-for-windows-10-binary-build.html

Related questions

Hadoop on Windows - "Error JAVA_HOME is incorrectly set."

I'm trying to get Hadoop running 'Local Mode' on my Windows machine. I've been using this guide: http://hadoop.apache.org/docs/r1.2.1/single_node_setup.html When I run the command: bin/hadoop, I get the following error message: …

Read More
Hadoop on windows server

I'm thinking about using hadoop to process large text files on my existing windows 2003 servers (about 10 quad core machines with 16gb of RAM) The questions are: Is there any good tutorial on how to configure an hadoop cluster on windows? …

Read More
How can you find out which process is listening on a port on Windows?

How can you find out which process is listening on a port on Windows?

Read More