How do I check for equality using Spark Dataframe without SQL Query?

scala apache-spark dataframe apache-spark-sql
Instinct picture Instinct · Jul 9, 2015 · Viewed 129.8k times · Source

I want to select a column that equals to a certain value. I am doing this in scala and having a little trouble.

Heres my code

df.select(df("state")==="TX").show()

this returns the state column with boolean values instead of just TX

Ive also tried

df.select(df("state")=="TX").show()

but this doesn't work either.

Answer

user3487888 picture user3487888 · Feb 8, 2016

I had the same issue, and the following syntax worked for me:

df.filter(df("state")==="TX").show()

I'm using Spark 1.6.

Related questions

Renaming column names of a DataFrame in Spark Scala

I am trying to convert all the headers / column names of a DataFrame in Spark-Scala. as of now I come up with following code which only replaces a single column name. for( i <- 0 to origCols.length - 1) { df.…

Read More
How to define partitioning of DataFrame?

I've started using Spark SQL and DataFrames in Spark 1.4.0. I'm wanting to define a custom partitioner on DataFrames, in Scala, but not seeing how to do this. One of the data tables I'm working with contains a list of transactions, …

Read More
Joining Spark dataframes on the key

I have constructed two dataframes. How can we join multiple Spark dataframes ? For Example : PersonDf, ProfileDf with a common column as personId as (key). Now how can we have one Dataframe combining PersonDf and ProfileDf?

Read More