How do I get Python libraries in pyspark?

python python-2.7 pyspark shapely
thenakulchawla picture thenakulchawla · Mar 25, 2016 · Viewed 44.8k times · Source

I want to use matplotlib.bblpath or shapely.geometry libraries in pyspark.

When I try to import any of them I get the below error:

>>> from shapely.geometry import polygon
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: No module named shapely.geometry

I know the module isn't present, but how can these packages be brought to my pyspark libraries?

Answer

armatita picture armatita · Mar 25, 2016

In the Spark context try using:

SparkContext.addPyFile("module.py")  # also .zip

, quoting from the docs:

Add a .py or .zip dependency for all tasks to be executed on this SparkContext in the future. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI.

Related questions

Iterating over dictionaries using 'for' loops

I am a bit puzzled by the following code: d = {'x': 1, 'y': 2, 'z': 3} for key in d: print key, 'corresponds to', d[key] What I don't understand is the key portion. How does Python recognize that it needs only to …

Read More
What is the result of % in Python?

What does the % in a calculation? I can't seem to work out what it does. Does it work out a percent of the calculation for example: 4 % 2 is apparently equal to 0. How?

Read More
How to fix: "UnicodeDecodeError: 'ascii' codec can't decode byte"

as3:~/ngokevin-site# nano content/blog/20140114_test-chinese.mkd as3:~/ngokevin-site# wok Traceback (most recent call last): File "/usr/local/bin/wok", line 4, in Engine() File "/usr/local/lib/python2.7/site-packages/wok/engine.py", line 104, in init self.load_pages() File "/usr/…

Read More