How to construct Dataframe from a Excel (xls,xlsx) file in Scala Spark?

excel scala apache-spark apache-spark-sql spark-excel
koiralo picture koiralo · May 26, 2017 · Viewed 81.6k times · Source

I have a large Excel(xlsx and xls) file with multiple sheet and I need convert it to RDD or Dataframe so that it can be joined to other dataframe later. I was thinking of using Apache POI and save it as a CSV and then read csv in dataframe. But if there is any libraries or API that can help in this Process would be easy. Any help is highly appreciated.

Answer

Ramesh Maharjan picture Ramesh Maharjan · May 26, 2017

The solution to your problem is to use Spark Excel dependency in your project.

Spark Excel has flexible options to play with.

I have tested the following code to read from excel and convert it to dataframe and it just works perfect

def readExcel(file: String): DataFrame = sqlContext.read
    .format("com.crealytics.spark.excel")
    .option("location", file)
    .option("useHeader", "true")
    .option("treatEmptyValuesAsNulls", "true")
    .option("inferSchema", "true")
    .option("addColorColumns", "False")
    .load()

val data = readExcel("path to your excel file")

data.show(false)

you can give sheetname as option if your excel sheet has multiple sheets

.option("sheetName", "Sheet2")

I hope its helpful

Related questions

How to unzip a zip file using scala?

Basically I need to unzip a .zip file which contains a folder called modeled which in turn contains a number of excel files. I have had some luck in finding code that was already written (ZipArchive) which is meant to …

Read More
Is there a Scala wrapper for Apache POI?

I would like to use Apache POI to read/create Excel files in an Scala app. Sure, I can use the POI library directly, it's Java after all, but I would like to have the Scala feel. So is there …

Read More
Is there a way to crack the password on an Excel VBA Project?

I've been asked to update some Excel 2003 macros, but the VBA projects are password protected, and it seems there's a lack of documentation... no-one knows the passwords. Is there a way of removing or cracking the password on a VBA …

Read More