How to add third-party Java JAR files for use in PySpark

python apache-spark pyspark py4j
WestCoastProjects picture WestCoastProjects · Dec 30, 2014 · Viewed 63.2k times · Source

I have some third-party database client libraries in Java. I want to access them through

java_gateway.py

E.g.: to make the client class (not a JDBC driver!) available to the Python client via the Java gateway:

java_import(gateway.jvm, "org.mydatabase.MyDBClient")

It is not clear where to add the third-party libraries to the JVM classpath. I tried to add to file compute-classpath.sh, but that did not seem to work. I get:

Py4jError: Trying to call a package

Also, when comparing to Hive: the hive JAR files are not loaded via file compute-classpath.sh, so that makes me suspicious. There seems to be some other mechanism happening to set up the JVM side classpath.

Answer

Marl picture Marl · Feb 12, 2015

You can add external jars as arguments to pyspark

pyspark --jars file1.jar,file2.jar

Related questions

findspark.init() IndexError: list index out of range error

When running the following in a Python 3.5 Jupyter environment I get the error below. Any ideas on what is causing it? import findspark findspark.init() Error: IndexError Traceback (most recent call last) <ipython-input-20-2ad2c7679ebc> in &…

Read More
Pyspark py4j PickleException: "expected zero arguments for construction of ClassDict"

This question is directed towards persons familiar with py4j - and can help to resolve a pickling error. I am trying to add a method to the pyspark PythonMLLibAPI that accepts an RDD of a namedtuple, does some work, …

Read More
Running custom Java class in PySpark

I'm trying to run a custom HDFS reader class in PySpark. This class is written in Java and I need to access it from PySpark, either from the shell or with spark-submit. In PySpark, I retrieve the JavaGateway from the …

Read More