What to set `SPARK_HOME` to?

python apache-spark pythonpath pyspark apache-zeppelin
A T picture A T · Jun 14, 2015 · Viewed 35.1k times · Source

Installed apache-maven-3.3.3, scala 2.11.6, then ran:

$ git clone git://github.com/apache/spark.git -b branch-1.4
$ cd spark
$ build/mvn -DskipTests clean package

Finally:

$ git clone https://github.com/apache/incubator-zeppelin
$ cd incubator-zeppelin/
$ mvn install -DskipTests

Then ran the server:

$ bin/zeppelin-daemon.sh start

Running a simple notebook beginning with %pyspark, I got an error about py4j not being found. Just did pip install py4j (ref).

Now I'm getting this error:

pyspark is not responding Traceback (most recent call last):
  File "/tmp/zeppelin_pyspark.py", line 22, in <module>
    from pyspark.conf import SparkConf
ImportError: No module named pyspark.conf

I've tried setting my SPARK_HOME to: /spark/python:/spark/python/lib. No change.

Answer

ChromeHearts picture ChromeHearts · Jun 14, 2015

Two environment variables are required:

SPARK_HOME=/spark
PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-VERSION-src.zip:$PYTHONPATH

Related questions

unable to add spark to PYTHONPATH

I am struggling to add spark to my python path: (myenv)me@me /home/me$ set SPARK_HOME="/home/me/spark-1.2.1-bin-hadoop2.4" (myenv)me@me /home/me$ set PYTHONPATH=$PYTHONPATH:$SPARK_HOME:$SPARK_HOME/python:$SPARK_HOME/python/build:$SPARK_…

Read More
How to change dataframe column names in pyspark?

I come from pandas background and am used to reading data from CSV files into a dataframe and then simply changing the column names to something useful using the simple command: df.columns = new_column_name_list However, the same …

Read More
Load CSV file with Spark

I'm new to Spark and I'm trying to read CSV data from a file with Spark. Here's what I am doing : sc.textFile('file.csv') .map(lambda line: (line.split(',')[0], line.split(',')[1])) .collect() I would expect …

Read More