How to start a Spark Shell using pyspark in Windows?

pyspark apache-spark-1.4
Alex picture Alex · Jul 28, 2015 · Viewed 16.8k times · Source

I am a beginner in Spark and trying to follow instructions from here on how to initialize Spark shell from Python using cmd: http://spark.apache.org/docs/latest/quick-start.html

But when I run in cmd the following:

C:\Users\Alex\Desktop\spark-1.4.1-bin-hadoop2.4\>c:\Python27\python bin\pyspark

then I receive the following error message:

File "bin\pyspark", line 21 
export SPARK_HOME="$(cd ="$(cd "`dirname "$0"`"/..; pwd)" 
SyntaxError: invalid syntax

What am I doing wrong here?

P.S. When in cmd I try just C:\Users\Alex\Desktop\spark-1.4.1-bin-hadoop2.4>bin\pyspark

then I receive ""python" is not recognized as internal or external command, operable program or batch file".

Answer

maxymoo picture maxymoo · Jul 28, 2015

You need to have Python available in the system path, you can add it with setx:

setx path "%path%;C:\Python27"

Related questions

How to change dataframe column names in pyspark?

I come from pandas background and am used to reading data from CSV files into a dataframe and then simply changing the column names to something useful using the simple command: df.columns = new_column_name_list However, the same …

Read More
Load CSV file with Spark

I'm new to Spark and I'm trying to read CSV data from a file with Spark. Here's what I am doing : sc.textFile('file.csv') .map(lambda line: (line.split(',')[0], line.split(',')[1])) .collect() I would expect …

Read More
How do I add a new column to a Spark DataFrame (using PySpark)?

I have a Spark DataFrame (using PySpark 1.5.1) and would like to add a new column. I've tried the following without any success: type(randomed_hours) # => list # Create in Python and transform to RDD new_col = pd.DataFrame(randomed_hours, …

Read More