How to spark-submit a python file in spark 2.1.0?

apache-spark pyspark apache-spark-sql pyspark-sql spark-submit
Kalyan picture Kalyan · Mar 23, 2017 · Viewed 17.3k times · Source

I am currently running spark 2.1.0. I have worked most of the time in PYSPARK shell, but I need to spark-submit a python file(similar to spark-submit jar in java) . How do you do that in python?

Answer

Himaprasoon picture Himaprasoon · Mar 23, 2017

pythonfile.py

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("appName").getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize([1,2,3,4,5,6,7])
print(rdd.count())

Run the above program with configurations you want : eg :

 YOUR_SPARK_HOME/bin/spark-submit --master yourSparkMaster --num-executors 20 \
        --executor-memory 1G --executor-cores 2 --driver-memory 1G \
        pythonfile.py

These options are not mandatory. You can even run like 

YOUR_SPARK_HOME/bin/spark-submit --master sparkMaster/local pythonfile.py

Related questions

Convert pyspark string to date format

I have a date pyspark dataframe with a string column in the format of MM-dd-yyyy and I am attempting to convert this into a date column. I tried: df.select(to_date(df.STRING_COLUMN).alias('new_date')).show() and …

Read More
Split Spark Dataframe string column into multiple columns

I've seen various people suggesting that Dataframe.explode is a useful way to do this, but it results in more rows than the original dataframe, which isn't what I want at all. I simply want to do the Dataframe equivalent …

Read More
How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?

import numpy as np df = spark.createDataFrame( [(1, 1, None), (1, 2, float(5)), (1, 3, np.nan), (1, 4, None), (1, 5, float(10)), (1, 6, float('nan')), (1, 6, float('nan'))], ('session', "timestamp1", "id2")) Expected output dataframe with count of nan/null for each column Note: The previous questions I found in stack overflow …

Read More