Apache Spark: How to use pyspark with Python 3

python python-3.x apache-spark
tchakravarty picture tchakravarty · May 16, 2015 · Viewed 83.5k times · Source

I built Spark 1.4 from the GH development master, and the build went through fine. But when I do a bin/pyspark I get the Python 2.7.9 version. How can I change this?

Answer

Rtik88 picture Rtik88 · Aug 19, 2015

Just set the environment variable:

export PYSPARK_PYTHON=python3

in case you want this to be a permanent change add this line to pyspark script.

Related questions

pyspark error does not exist in the jvm error when initializing SparkContext

I am using spark over emr and writing a pyspark script, I am getting an error when trying to from pyspark import SparkContext sc = SparkContext() this is the error File "pyex.py", line 5, in <module> sc = SparkContext() File "/…

Read More
Convert bytes to a string

I'm using this code to get standard output from an external program: >>> from subprocess import * >>> command_stdout = Popen(['ls', '-l'], stdout=PIPE).communicate()[0] The communicate() method returns an array of bytes: >>> …

Read More
Best way to convert string to bytes in Python 3?

There appear to be two different ways to convert a string to bytes, as seen in the answers to TypeError: 'str' does not support the buffer interface Which of these methods would be better or more Pythonic? Or is it …

Read More