Mock a Spark RDD in the unit tests

scala unit-testing mocking apache-spark scalatest
Edamame picture Edamame · Jun 19, 2015 · Viewed 14.4k times · Source

Is it possible to mock a RDD without using sparkContext?

I want to unit test the following utility function:

 def myUtilityFunction(data1: org.apache.spark.rdd.RDD[myClass1], data2: org.apache.spark.rdd.RDD[myClass2]): org.apache.spark.rdd.RDD[myClass1] = {...}

So I need to pass data1 and data2 to myUtilityFunction. How can I create a data1 from a mock org.apache.spark.rdd.RDD[myClass1], instead of create a real RDD from SparkContext? Thank you!

Answer

Holden picture Holden · Jun 19, 2015

RDDs are pretty complex, mocking them is probably not the best way to go about creating test data. Instead I'd recommend using sc.parallelize with your data. I'm also (somewhat biased) think that https://github.com/holdenk/spark-testing-base can help by providing a trait to setup & teardown the Spark Context for your tests.

Related questions

How to write unit tests in Spark 2.0+?

I've been trying to find a reasonable way to test SparkSession with the JUnit testing framework. While there seem to be good examples for SparkContext, I couldn't figure out how to get a corresponding example working for SparkSession, even though …

Read More
What’s the difference between ScalaTest and Scala Specs unit test frameworks?

Both are BDD (Behavior Driven Development) capable unit test frameworks for Scala written in Scala. And Specs is built upon may also involve the ScalaTest framework. But what does Specs offer ScalaTest doesn't? What are the differences?

Read More
Comparing collection contents with ScalaTest

I'm trying to unit-test some Scala that is very collection-heavy. These collections are returned as Iterable[T], so I am interested in the contents of the collection, even if the underlying types differ. This is actually two related problems: How …

Read More