How to install Apache Spark on HortonWorks HDP 2.2 (built using Ambari)

hortonworks-data-platform ambari
Knows Not Much picture Knows Not Much · Dec 23, 2014 · Viewed 7.3k times · Source

I successfully built a 5 node cluster of HortonWorks HDP 2.2 using Ambari.

However I don't see Apache Spark in the installed services list.

I did some research and found that Ambari does not install certain components like hue etc. ( Spark was not in that list, but I guess its not installed).

How do I do a manual install of Apache spark on my 5 node HDP 2.2?

Or should I delete my cluster and perform a fresh install without using Ambari?

Answer

philshem picture philshem · Dec 23, 2014

Hortonworks support for Spark is arriving but not fully complete (details and blog).

Instructions for how to integrate Spark with HDP can be found here.

Related questions

How to disable Transparent Huge Pages (THP) in Ubuntu 16.04LTS

I am setting up an ambari cluster with 3 virtualbox VMs running Ubuntu 16.04LTS. However I get the below warning: The following hosts have Transparent Huge Pages (THP) enabled. THP should be disabled to avoid potential Hadoop performance issues. How can …

Read More
Spark read file from S3 using sc.textFile ("s3n://...)

Trying to read a file located in S3 using spark-shell: scala> val myRdd = sc.textFile("s3n://myBucket/myFile1.log") lyrics: org.apache.spark.rdd.RDD[String] = s3n://myBucket/myFile1.log MappedRDD[55] at textFile at <console>:12 …

Read More
How to delete files from the HDFS?

I just downloaded Hortonworks sandbox VM, inside it there are Hadoop with the version 2.7.1. I adding some files by using the hadoop fs -put /hw1/* /hw1 ...command. After it I am deleting the added files, by the hadoop fs -rm /…

Read More