Number of partitions in RDD and performance in Spark

performance apache-spark pyspark rdd
mar tin picture mar tin · Mar 4, 2016 · Viewed 34.2k times · Source

In Pyspark, I can create a RDD from a list and decide how many partitions to have:

sc = SparkContext()
sc.parallelize(xrange(0, 10), 4)

How does the number of partitions I decide to partition my RDD in influence the performance? And how does this depend on the number of core my machine has?

Answer

StephenBoesch picture StephenBoesch · Mar 4, 2016

The primary effect would be by specifying too few partitions or far too many partitions.

Too few partitions You will not utilize all of the cores available in the cluster.

Too many partitions There will be excessive overhead in managing many small tasks.

Between the two the first one is far more impactful on performance. Scheduling too many smalls tasks is a relatively small impact at this point for partition counts below 1000. If you have on the order of tens of thousands of partitions then spark gets very slow.

Related questions

Spark performance for Scala vs Python

I prefer Python over Scala. But, as Spark is natively written in Scala, I was expecting my code to run faster in the Scala than the Python version for obvious reasons. With that assumption, I thought to learn & write …

Read More
Spark functions vs UDF performance?

Spark now offers predefined functions that can be used in dataframes, and it seems they are highly optimized. My original question was going to be on which is faster, but I did some testing myself and found the spark functions …

Read More
Apache Spark: map vs mapPartitions?

What's the difference between an RDD's map and mapPartitions method? And does flatMap behave like map or like mapPartitions? Thanks. (edit) i.e. what is the difference (either semantically or in terms of execution) between def map[A, B](rdd: …

Read More