Add Number of days column to Date Column in same dataframe for Spark Scala App

scala apache-spark dataframe dateadd
qubiter picture qubiter · Jun 5, 2017 · Viewed 13.1k times · Source

I have a dataframe df of columns ("id", "current_date", "days") and I am trying to add the the "days" to "current_date" and create a new dataframe with new column called "new_date" using spark scala function date_add()

val newDF = df.withColumn("new_Date", date_add(df("current_date"), df("days").cast("Int")))

But looks like the function date_add only accepts Int values and not columns. How can get the desired output in such case? Are there any alternative functions i can use to get the desired output?

spark version: 1.6.0 scala version: 2.10.6

Answer

Raphael Roth picture Raphael Roth · Jun 5, 2017

No need to use an UDF, you can do it using an SQL expression:

val newDF = df.withColumn("new_date", expr("date_add(current_date,days)"))

Related questions

Renaming column names of a DataFrame in Spark Scala

I am trying to convert all the headers / column names of a DataFrame in Spark-Scala. as of now I come up with following code which only replaces a single column name. for( i <- 0 to origCols.length - 1) { df.…

Read More
How to define partitioning of DataFrame?

I've started using Spark SQL and DataFrames in Spark 1.4.0. I'm wanting to define a custom partitioner on DataFrames, in Scala, but not seeing how to do this. One of the data tables I'm working with contains a list of transactions, …

Read More
Joining Spark dataframes on the key

I have constructed two dataframes. How can we join multiple Spark dataframes ? For Example : PersonDf, ProfileDf with a common column as personId as (key). Now how can we have one Dataframe combining PersonDf and ProfileDf?

Read More