How to reverse ordering for RDD.takeOrdered()?

StackG picture StackG · Oct 15, 2014 · Viewed 17.4k times · Source

What is the syntax to reverse the ordering for the takeOrdered() method of an RDD in Spark?

For bonus points, what is the syntax for custom-ordering for an RDD in Spark?

Answer

gasparms picture gasparms · Oct 16, 2014

Reverse Order

val seq = Seq(3,9,2,3,5,4)
val rdd = sc.parallelize(seq,2)
rdd.takeOrdered(2)(Ordering[Int].reverse)

Result will be Array(9,5)

Custom Order

We will sort people by age.

case class Person(name:String, age:Int)
val people = Array(Person("bob", 30), Person("ann", 32), Person("carl", 19))
val rdd = sc.parallelize(people,2)
rdd.takeOrdered(1)(Ordering[Int].reverse.on(x=>x.age))

Result will be Array(Person(ann,32))