Spark Implicit $ for DataFrame

Ben Nelson picture Ben Nelson · Feb 13, 2017 · Viewed 13.4k times · Source

Where in the sqlContext.implicits._ does it define the $"string" to represent a dataframe call to the parent dataframe's column? Specifically I was confused on seeing something like the following:

import sqlContext.implicits._
df.where($"type".isin("type1","type2") and $"status".isin("completed","inprogress"))

Answer

user2844511 picture user2844511 · Feb 13, 2017

If you see the following link

class SQLContext.implicits$.StringToColumn

Converts $"col name" into an Column.

Here is the link for latest version.