How to Use both Scala and Python in a same Spark project?

python scala apache-spark pyspark spark-streaming
Wilson Liao picture Wilson Liao · Oct 6, 2015 · Viewed 19.4k times · Source

Is that possible to pipe Spark RDD to Python?

Because I need a python library to do some calculation on my data, but my main Spark project is based on Scala. Is there a way to mix them both or let python access the same spark context?

Answer

Stephen De Gennaro picture Stephen De Gennaro · Oct 6, 2015

You can indeed pipe out to a python script using Scala and Spark and a regular Python script.

test.py

#!/usr/bin/python

import sys

for line in sys.stdin:
  print "hello " + line

spark-shell (scala)

val data = List("john","paul","george","ringo")

val dataRDD = sc.makeRDD(data)

val scriptPath = "./test.py"

val pipeRDD = dataRDD.pipe(scriptPath)

pipeRDD.foreach(println)

Output

hello john

hello ringo

hello george

hello paul

Related questions

How to turn off INFO logging in Spark?

I installed Spark using the AWS EC2 guide and I can launch the program fine using the bin/pyspark script to get to the spark prompt and can also do the Quick Start quide successfully. However, I cannot for the …

Read More
How to use JDBC source to write and read data in (Py)Spark?

The goal of this question is to document: steps required to read and write data using JDBC connections in PySpark possible issues with JDBC sources and know solutions With small changes these methods should work with other supported languages including …

Read More
How do I read a parquet in PySpark written from Spark?

I am using two Jupyter notebooks to do different things in an analysis. In my Scala notebook, I write some of my cleaned data to parquet: partitionedDF.select("noStopWords","lowerText","prediction").write.save("swift2d://xxxx.keystone/commentClusters.parquet") I …

Read More