jdbc.SQLServerException: The "variant" data type is not supported

python sql pyspark mssql-jdbc
Renos Bardhis picture Renos Bardhis · Sep 12, 2019 · Viewed 7.4k times · Source

I use pyspark 2.4.4 when I am getting the data from SQL Server I get com.microsoft.sqlserver.jdbc.SQLServerException: The "variant" data type is not supported.

What is needed to do? to fix this problem. I read this. But, I do not understand.

EDIT: The following is the code I use to create the error message: 

for i in df['name']: mssql_df = spark.read.format("jdbc") \ 
.option("url", "jdbc:sqlserver://SERVERNAME:1433;databaseName=DB;integratedSecurity=true") \ 
.option("dbtable", "[" + i + "]") \ 
.option("driver", 'com.microsoft.sqlserver.jdbc.SQLServerDriver').load()

Answer

Derek Mwachinga picture Derek Mwachinga · Dec 12, 2019

This answer comes a bit late but might be of help to future searches for solution of that error. I simply upgraded my connector from version 6.1.0.jre8 to 8.1.0.jre8-preview and the error was resolved.

You can check for latest version in Maven Repository.

Related questions

How to build a sparkSession in Spark 2.0 using pyspark?

I just got access to spark 2.0; I have been using spark 1.6.1 up until this point. Can someone please help me set up a sparkSession using pyspark (python)? I know that the scala examples available online are similar (here), but I …

Read More
Filtering a Pyspark DataFrame with SQL-like IN clause

I want to filter a Pyspark DataFrame with a SQL-like IN clause, as in sc = SparkContext() sqlc = SQLContext(sc) df = sqlc.sql('SELECT * from my_df WHERE field1 IN a') where a is the tuple (1, 2, 3). I am getting this error: …

Read More
How to use window functions in PySpark?

I'm trying to use some windows functions (ntile and percentRank) for a data frame but I don't know how to use them. Can anyone help me with this please? In the Python API documentation there are no examples about it. …

Read More