How to convert Row of a Scala DataFrame into case class most efficiently?
Once I have got in Spark some Row class, either Dataframe or Catalyst, I want to convert it to a case class in my code. This can be done by matching
someRow match {case Row(a:Long,b:String,c:Double) => myCaseClass(a,b,c)}
But it becomes ugly when the row has a huge number of columns, say a dozen of Doubles, some Booleans and even the occasional null.
I would like just to be able to -sorry- cast Row to myCaseClass. Is it possible, or have I already got the most economical syntax?
Answer
DataFrame is simply a type alias of Dataset[Row] . These operations are also referred as “untyped transformations” in contrast to “typed transformations” that come with strongly typed Scala/Java Datasets.
The conversion from Dataset[Row] to Dataset[Person] is very simple in spark
val DFtoProcess = SQLContext.sql("SELECT * FROM peoples WHERE name='test'")
At this point, Spark converts your data into DataFrame = Dataset[Row], a collection of generic Row object, since it does not know the exact type.
// Create an Encoders for Java class (In my eg. Person is a JAVA class)
// For scala case class you can pass Person without .class reference
val personEncoder = Encoders.bean(Person.class)
val DStoProcess = DFtoProcess.as[Person](personEncoder)
Now, Spark converts the Dataset[Row] -> Dataset[Person] type-specific Scala / Java JVM object, as dictated by the class Person.
Please refer to below link provided by databricks for further details