Left Anti join in Spark dataframes

scala apache-spark anti-join
Alejandro Martinez Otal picture Alejandro Martinez Otal · Jul 25, 2018 · Viewed 9.6k times · Source

I have two dataframes, and I would like to retrieve only the information of one of the dataframes, which is not found in the inner join, see the picture:

Full outer join

I have tried several ways: Inner join and filtering the rows that return at least one null, all the types of joins described on the docs for Spark 1.6 but failed to do obtain the results from just one join.

Can anybody help?

Answer

Manoj Kumar Dhakad picture Manoj Kumar Dhakad · Jul 25, 2018

This is called right excluding join and you can do like below

df1.join(df2,df1("column1")===df2("column2"),"right_outer").filter("column1 is null").show

Related questions

How can I change column types in Spark SQL's DataFrame?

Suppose I'm doing something like: val df = sqlContext.load("com.databricks.spark.csv", Map("path" -> "cars.csv", "header" -> "true")) df.printSchema() root |-- year: string (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to convert rdd object to dataframe in spark

How can I convert an RDD (org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]) to a Dataframe org.apache.spark.sql.DataFrame. I converted a dataframe to rdd using .rdd. After processing it I want it back in …

Read More