Spark-sqlserver connection

sql-server apache-spark data-analysis
Tia picture Tia · Jan 17, 2018 · Viewed 11.1k times · Source

Can we connect spark with sql-server? If so, how? I am new to spark, I want to connect the server to spark and work directly from sql-server instead of uploading .txt or .csv file. Please help, Thank you.

Answer

Ajay Kharade picture Ajay Kharade · Oct 29, 2018 
// Spark 2.x
import org.apache.spark.SparkContext

// Create dataframe on top of SQLServer database table
val sqlContext = new org.apache.spark.sql.SQLContext(sc)

val jdbcDF = sqlContext.read.format("jdbc").option("driver" , "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
           .option("url", "jdbc:sqlserver://XXXXX.com:port;databaseName=xxx") \
           .option("dbtable", "(SELECT * FROM xxxx) tmp") \
           .option("user", "xxx") \
           .option("password", "xxx") \
           .load()

// show sample records from data frame

jdbcDF.show(5)

Related questions

How do I UPDATE from a SELECT in SQL Server?

In SQL Server, it is possible to insert rows into a table with an INSERT.. SELECT statement: INSERT INTO Table (col1, col2, col3) SELECT col1, col2, col3 FROM other_table WHERE sql = 'cool' Is it also possible to update a …

Read More
How do I perform an IF...THEN in an SQL SELECT?

How do I perform an IF...THEN in an SQL SELECT statement? For example: SELECT IF(Obsolete = 'N' OR InStock = 'Y' ? 1 : 0) AS Saleable, * FROM Product

Read More
OR is not supported with CASE Statement in SQL Server

The OR operator in the WHEN clause of a CASE statement is not supported. How can I do this? CASE ebv.db_no WHEN 22978 OR 23218 OR 23219 THEN 'WECS 9500' ELSE 'WECS 9520' END as wecs_system

Read More