How to cast a WrappedArray[WrappedArray[Float]] to Array[Array[Float]] in spark (scala)

arrays scala casting spark-dataframe apache-spark-2.0
bobo32 picture bobo32 · Jan 28, 2017 · Viewed 13.8k times · Source

Im using Spark 2.0. I have a column of my dataframe containing a WrappedArray of WrappedArrays of Float.

An example of a row would be:

[[1.0 2.0 2.0][6.0 5.0 2.0][4.0 2.0 3.0]]

Im trying to transform this column into an Array[Array[Float]].

What I tried so far is the following:

dataframe.select("mycolumn").rdd.map(r => r.asInstanceOf[Array[Array[Float]]])

but I get the following error:

Caused by: java.lang.ClassCastException:
 org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to [[F

Any idea would be highly appreciated. Thanks

Answer

Sami Badawi picture Sami Badawi · Jan 28, 2017

Try this:

  val wawa: WrappedArray[WrappedArray[Float]] = null 
  val res: Array[Array[Float]] = wawa.map(inner => inner.array).toArray

It compiles for me

Related questions

Appending an element to the end of a list in Scala

I can't add an element of type T into a list List[T]. I tried with myList ::= myElement but it seems it creates a strange object and accessing to myList.last always returns the first element that was put inside …

Read More
Scala: what is the best way to append an element to an Array?

Say I have an Array[Int] like val array = Array( 1, 2, 3 ) Now I would like to append an element to the array, say the value 4, as in the following example: val array2 = array + 4 // will not compile I can of course use …

Read More
Difference between Array and List in scala

In what cases I should use Array(Buffer) and List(Buffer). Only one difference that I know is that arrays are nonvariant and lists are covariant. But what about performance and some other characteristics?

Read More