ModuleNotFoundError: No module named 'py4j'

python-3.x apache-spark hadoop pyspark py4j
Jassim Elakrouch picture Jassim Elakrouch · May 28, 2019 · Viewed 8.6k times · Source

I installed Spark and I am running into problems loading the pyspark module into ipython. I'm getting the following error:

Answer

RayZen picture RayZen · Jul 30, 2019

If you can run spark directly, maybe you have to fix the environment variable PYTHONPATH. Check the filename in the directory $SPARK_HOME/python/lib/. If the Spark version 2.4.3, the file is py4j-0.10.7-src.zip:

export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH

Related questions

Apache Spark: How to use pyspark with Python 3

I built Spark 1.4 from the GH development master, and the build went through fine. But when I do a bin/pyspark I get the Python 2.7.9 version. How can I change this?

Read More
py4j.protocol.Py4JJavaError occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe

I installed apache-spark and pyspark on my machine (Ubuntu), and in Pycharm, I also updated the environment variables (e.g. spark_home, pyspark_python). I'm trying to do: import os, sys os.environ['SPARK_HOME'] = ".../spark-2.3.0-bin-hadoop2.7" sys.path.append(".../…

Read More
pyspark error does not exist in the jvm error when initializing SparkContext

I am using spark over emr and writing a pyspark script, I am getting an error when trying to from pyspark import SparkContext sc = SparkContext() this is the error File "pyex.py", line 5, in <module> sc = SparkContext() File "/…

Read More