What is the Scala type mapping for all Spark SQL DataType

sql scala apache-spark apache-spark-sql sqldatatypes
aa8y picture aa8y · Oct 2, 2015 · Viewed 10.7k times · Source

The different DataTypes available for Spark SQL can be found here. Can anyone please tell me what would be the corresponding Java/Scala data type for each of Spark SQL's DataTypes?

Answer

zero323 picture zero323 · Oct 2, 2015

Directly from the Spark SQL and DataFrame Guide:

Data type       |    Value type in Scala
------------------------------------------------
ByteType        |    Byte   
ShortType       |    Short  
IntegerType     |    Int    
LongType        |    Long   
FloatType       |    Float  
DoubleType      |    Double     
DecimalType     |    java.math.BigDecimal
StringType      |    String
BinaryType      |    Array[Byte]
BooleanType     |    Boolean 
TimestampType   |    java.sql.Timestamp
DateType        |    java.sql.Date
ArrayType       |    scala.collection.Seq   
MapType         |    scala.collection.Map   
StructType      |    org.apache.spark.sql.Row

Related questions

How to select the first row of each group?

I have a DataFrame generated as follow: df.groupBy($"Hour", $"Category") .agg(sum($"value") as "TotalValue") .sort($"Hour".asc, $"TotalValue".desc)) The results look like: +----+--------+----------+ |Hour|Category|TotalValue| +----+--------+----------+ | 0| cat26| 30.9| | 0| cat13| 22.1| | 0| cat95| 19.6| | 0| cat105| 1.3| | 1| cat67| 28.5| | 1| cat4| 26.8| | 1| …

Read More
Querying Spark SQL DataFrame with complex types

How Can I query an RDD with complex types such as maps/arrays? for example, when I was writing this test code: case class Test(name: String, map: Map[String, String]) val map = Map("hello" -> "world", "hey" -&…

Read More
Including null values in an Apache Spark Join

I would like to include null values in an Apache Spark join. Spark doesn't include rows with null by default. Here is the default Spark behavior. val numbersDf = Seq( ("123"), ("456"), (null), ("") ).toDF("numbers") val lettersDf = Seq( ("123", "abc"), ("456", "def"), (null, "zzz"), ("", "hhh") ).…

Read More