how to create DataFrame from multiple arrays in Spark Scala?

arrays scala linear-regression spark-dataframe
Sam picture Sam · May 11, 2016 · Viewed 19.6k times · Source 
val tvalues: Array[Double] = Array(1.866393526974307, 2.864048126935307, 4.032486069215076, 7.876169953355888, 4.875333799256043, 14.316322626848278)
val pvalues: Array[Double] = Array(0.064020056478447, 0.004808399479386827, 8.914865448939047E-5, 7.489564524121306E-13, 2.8363794106756046E-6, 0.0)

I have two Arrays as above, i need to build a DataFrame from this Arrays like the following,

Tvalues                Pvalues
1.866393526974307      0.064020056478447
2.864048126935307      0.004808399479386827
......                 .....

As of now i am trying with StringBuilder in Scala. which doesnt go as expected. help me on this please.

Answer

elm picture elm · May 11, 2016

Try for instance

val df = sc.parallelize(tpvalues zip pvalues).toDF("Tvalues","Pvalues")

and thus 

scala> df.show
+------------------+--------------------+
|          Tvalues|             Pvalues|
+------------------+--------------------+
| 1.866393526974307|   0.064020056478447|
| 2.864048126935307|0.004808399479386827|
| 4.032486069215076|8.914865448939047E-5|
| 7.876169953355888|7.489564524121306...|
| 4.875333799256043|2.836379410675604...|
|14.316322626848278|                 0.0|
+------------------+--------------------+

Using parallelize we obtain an RDD of tuples -- the first element from the first array, the second element from the other array --, which is transformed into a dataframe of rows, one row for each tuple.

Update

For dataframe'ing multiple arrays (all with the same size), for instance 4 arrays, consider

case class Row(i: Double, j: Double, k: Double, m: Double)

val xs = Array(arr1, arr2, arr3, arr4).transpose
val rdd = sc.parallelize(xs).map(ys => Row(ys(0), ys(1), ys(2), ys(3))
val df = rdd.toDF("i","j","k","m")

Related questions

Appending an element to the end of a list in Scala

I can't add an element of type T into a list List[T]. I tried with myList ::= myElement but it seems it creates a strange object and accessing to myList.last always returns the first element that was put inside …

Read More
Scala: what is the best way to append an element to an Array?

Say I have an Array[Int] like val array = Array( 1, 2, 3 ) Now I would like to append an element to the array, say the value 4, as in the following example: val array2 = array + 4 // will not compile I can of course use …

Read More
Difference between Array and List in scala

In what cases I should use Array(Buffer) and List(Buffer). Only one difference that I know is that arrays are nonvariant and lists are covariant. But what about performance and some other characteristics?

Read More