Spark: subtract two DataFrames

apache-spark dataframe rdd
Interfector picture Interfector · Apr 9, 2015 · Viewed 110.7k times · Source

In Spark version 1.2.0 one could use subtract with 2 SchemRDDs to end up with only the different content from the first one

val onlyNewData = todaySchemaRDD.subtract(yesterdaySchemaRDD)

onlyNewData contains the rows in todaySchemRDD that do not exist in yesterdaySchemaRDD.

How can this be achieved with DataFrames in Spark version 1.3.0?

Answer

Eric Eijkelenboom picture Eric Eijkelenboom · Apr 10, 2015

According to the api docs, doing: 

dataFrame1.except(dataFrame2)

will return a new DataFrame containing rows in dataFrame1 but not in dataframe2.

Related questions

How to create a DataFrame from a text file in Spark

I have a text file on HDFS and I want to convert it to a Data Frame in Spark. I am using the Spark Context to load the file and then try to generate individual columns from that file. val …

Read More
Difference between DataFrame, Dataset, and RDD in Spark

I'm just wondering what is the difference between an RDD and DataFrame (Spark 2.0.0 DataFrame is a mere type alias for Dataset[Row]) in Apache Spark? Can you convert one to the other?

Read More
Scala Spark : How to create a RDD from a list of string and convert to DataFrame

I want to create a DataFrame from a list of string that could match existing schema. Here is my code. val rowValues = List("ann", "f", "90", "world", "23456") // fails val rowValueTuple = ("ann", "f", "90", "world", "23456") //works val newRow = sqlContext.sparkContext.parallelize(Seq(rowValueTuple)).…

Read More