Understanding Kryo serialization buffer overflow error

scala apache-spark kryo
soupybionics picture soupybionics · Apr 1, 2018 · Viewed 8.7k times · Source

I am trying to understand the following error and I am running in client ode.

 org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 61186304. To avoid this, increase spark.kryoserializer.buffer.max value.
        at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:300)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:313)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Basically I am trying to narrow down the problem. Is my understanding right that this error is occurring in the spark driver side(i am on aws emr so I believe this will be running on master)? and I should be looking at spark.driver.memory ?

Answer

Mike Pone picture Mike Pone · Feb 19, 2019

No, the problem is that kryo does not have enough room in its buffer. You should be adjusting spark.kryoserializer.buffer.max in your properties file, or use --conf "spark.kryoserializer.buffer.max=128m" in your spark-submit command. 128m should be big enough for you.

Related questions

Customize SparkContext using sparkConf.set(..) when using spark-shell

In Spark, there are 3 primary ways to specify the options for the SparkConf used to create the SparkContext: As properties in the conf/spark-defaults.conf e.g., the line: spark.driver.memory 4g As args to spark-shell or spark-submit e.…

Read More
When to use Kryo serialization in Spark?

I am already compressing RDDs using conf.set("spark.rdd.compress","true") and persist(MEMORY_AND_DISK_SER). Will using Kryo serialization make the program even more efficient, or is it not useful in this case? I know that Kryo …

Read More
Spark Kryo: Register a custom serializer

I have a class that implements a custom Kryo serializer by implementing the read() and write() methods from com.esotericsoftware.kryo.Serializer (see example below). How can I register this custom serializer with Spark? Here is a pseudo-code example of …

Read More